Load in the Loop: Episode 4 - Groms

Load In the Loop: Episode 4 is here! Join Eric Hileman and Ivan Chepurnyi as they identify and fix some performance issues for Groms.com, a beautifully designed furniture website.

Ivan's Talk on Magento Architecture

Follow us on Twitter! @loadintheloop

Sponsorship provided by blackfire.io

Contact info: 
Ivan: ivan.chepurnyi@ecomdev.org, @IvanChepurnyi
Eric: litl@magemojo.com, @ericvhileman

Transcript

Episode 4

Eric Hileman: [00:00:00] This is Load in the Loop sponsored by Blackfire IO. I'm Eric, the CEO and co founder of MageMojo Magento hosting. For the last 10 years, we've been hosting exclusively Magento and we provide a deep level of support for Magento. On this podcast,  we review some of the customers and we share what we find to help others.

I'm here with my cohost, Ivan.

Ivan Chepurnyi: [00:00:17] Hello. My name is Ivan, I'm a Magento performance expert. I live in the Netherlands, run my own business independently without any team behind me. Just me going to merchants and helping them out with solving performance issues. Unfortunately, I can't clone myself yet, and it's the only way how I can work right now because I don't have time to invest into training an employee of mine. , to do the same job as I do, or the people who have the same level of experience as me are too expensive for me to hire? So here we are in the show Load in the Loop and, um, today, um, we have a very interesting store to review. Um. I really like it. So Eric, can you tell a little about the merchant?

Eric Hileman: [00:01:14] Yeah. The merchant's Groms. They're here in Brooklyn and they sell furniture, so they have a really nice site design here and they're getting ready for the holidays, they do run sales, you know, probably not as aggressive as everybody else does, but they just, they put this new design together and it looks good and they need some, some additional advice on what's going on under the hood with some of the performance problems that they're seeing.

So...

Ivan Chepurnyi: [00:01:41] They know that, um, Magento's first demo store has been mostly about furniture.

Eric Hileman: [00:01:47] Yeah, that's true.

Ivan Chepurnyi: [00:01:50] I still remember the times when I was  working on Magento one core and we had to fix bugs and every single installment had actually furniture over there, and that's how I learned the names in English for furniture like ottoman, sofa, and other stuff. It was quite a fun experience, but after some time, you know, you get so much tired of furniture, but this website looks amazing like... The design and approach they took. It's so beautiful. I could work it on this Magento store for a day, you know, like and still be enjoyed with the way how it's built. It's so visually pleasing, in my opinion.

Eric Hileman: [00:02:33] Cool. I think,  , so they've been having some problems on not so much the homepage, but into the, um, into the category page. And of course, the, you know, product display page.

Let's see. It's using a lot more CPU than it should, which is the usual suspect.

Ivan Chepurnyi: [00:02:54] And also you know, like during rest of our episodes, people just leaving some comments and they telll, like, they also interested to learn a little bit about you know, not only about Blackfire profiling, but also, um you know, things like New Relic putting , you know, abnormalities in New Relic and other stuff.

Maybe we should go over it in the future additions, but I think with New Relic data it's a little bit, um, I think it's going to be little bit harder to obtain for an actual show because, you know, just profiling a simple page.

Eric Hileman: [00:03:34] Yeah, different tools, definitely different tools.

Ivan Chepurnyi: [00:03:38] And also New Relic might have some data that, is not meant to be shown, you know, publicly. You know,because if it has some information about the visitors, you know, so it's not so, ...

We can talk a little bit. Um, maybe we can stop some demo store and show with the New Relic? What do we look for?

Eric Hileman: [00:04:00] It would be, it would be nice to come back around and show some of the sites that we've helped New Relic before and after because that that graph of CPU is incredible. Like from the sites that we've done and the changes that you've recommended and they've implemented, it's just the graph just drops off with resource utilizations.

It's so beautiful.

Ivan Chepurnyi: [00:04:20] Yeah. Pity that for fireplace actually, they don't have history before. , our changes. So it would be so interesting to see. And yeah. So, um, you probably already see my screen, right? Yup. So here is our current design. I, I even don't want to press Blackfire profile button, it's  just so pleasing.

Okay. So let's just start profiling homepage. Shall I disable aggregation? Yeah. As a first page, we wouldn't, we don't know what's going to happen there so I just disable aggregation. And probably it makes sense to enable aggregation when we really have some kind of... Oh, it was quite fast. Yeah. I wouldn't have had time, you know, check some stuff so let's see what it's going to show us. So it was five seconds. Okay. It was five, five, a fast five seconds. All right, so for the home page, 360 database queries, it's a little bit too much I can tell you for sure. So let's dive yeah. Let's, first of all, take a look what's shown on the page. So there is some content blocks with some categories?

Okay, and also we have some products showcased on the homepage as well. It's like probably like 15 products or something like that. So let's take a look at the timeline and see where those four seconds are spent on. So what we see here is actually, there is one big block. , over here that takes 3 and half seconds, right?

It's called category slider. And I already see that there is some kind of widget data. , so this get widget product data. This one takes three seconds. And I already see here a familiar pattern, for load in the loops. So yeah.

Yep. So here is, we are seeing actually, is that a, there is some kind of product load? Oh yeah. So it uses get product repository, get by ID.

Eric Hileman: [00:07:05] That's a common offender, right?

Ivan Chepurnyi: [00:07:07] Yeah. So this is a something that's happening, and probably it happens for every single product that is decided to be loaded over there and

Eric Hileman: [00:07:20] that loads the full full product.

Ivan Chepurnyi: [00:07:23] And there is already a collection being used? Just take a look here. So here we see that probably this widget get product data. First of all, it obtains a collection of products loads them, and then iterates over every single product in this collection.

Eric Hileman: [00:07:41] Is this, widget built in?

Is this a built in widget?

Ivan Chepurnyi: [00:07:44] , looks like not.

Eric Hileman: [00:07:46] Oh, okay. Okay.

Ivan Chepurnyi: [00:07:48] So it's probably made custom for a product. Okay. , for this project. So what I see here is that, . Yeah. There is just full product load in the loop happening.

Eric Hileman: [00:08:02] Yeah, it's that product repository as usual. So how does, how would they get around that then?

They already have the collection loaded.

Ivan Chepurnyi: [00:08:09] Yeah. They already have a collection loaded. I know for which purpose they access a product separately. Maybe it's a, a, they don't have some attributes available or something like that.

Eric Hileman: [00:08:19] Oh, like the price, like did they have a sale price there? So like, is that

Ivan Chepurnyi: [00:08:24] a, maybe it's, they have some kind of custom attributes, but it makes sense just to add this attribute to the collection load. Collection has a message called add attribute to select and this message can be invoked in order to get the data that you need from product within a collection, without, you know, working over every single product and loading it separately.

Yeah. It's, it's very pleasing. There is not much product to worry about here, I think. But, you know, like you can save three seconds if you just use regular collection over here.

Eric Hileman: [00:09:10] That would be huge.

Ivan Chepurnyi: [00:09:13] Yeah. Like 80% of the load time. Yeah. And here if we opened tdatabase queries, yeah, we see, we see is that also is there is configurable products as well.

A bunch of things are loaded in the loop. So basically everything that we see here, like everything that looks 17 andabove. That's all a result of load in the loop. So as soon as only collection going to be used, um, there is definitely going to be much more or less database queries, there is still something that happens in default Magento core.

When you let's say. , access the price for a configurable, Magento still,  loads, um, a price in the loop by loading all of those simple products for configurable and then for every single simple product, it also does a database query to tier price. So here, for instance, you see.

, is there a query to retrieve tier price of a product, um, but that the one actually can be reduced just by delaying some very simple implementation with using data directly available in the index offer price and this price index information is already available in collection.

, you have to do just couple of plugins. You have to build on top of the price box, Magento standard price box in order to utilize the data from the index instead of using full resolution of the product information to obtain the price of the configurable or price of a simple product. And also it doesn't make sense to check tier prices where you are just going to show the product on the product listing.

So yeah, in general, this one can be really reduced and also you see the time spent on database queries, is quite a lot as well. So it's almost a hundred milliseconds and some of the most interesting ones are this one like four, forty.  Yeah, it's a, this is a table.

Um, this is a table used by Magento, um, in order to create relations between simple and configurable products.

Eric Hileman: [00:11:53] , ah, okay. Okay.

Ivan Chepurnyi: [00:11:55] , the fun fact the name super comes from the Magento one era. So when we were developing this functionality in Magento core in Magento One era it was not called configurable product.

It was called super product. But then at the last moment. , it was decided, Hey, actually no one understands what is a super product. So let's rename it to configurable product.

Eric Hileman: [00:12:22] Marketing team can't sell that.

Ivan Chepurnyi: [00:12:26] So basically there is a lot of stuff like that in Magento core where as the name of the table does not correspond to the actual name of the functionality. So yeah, just one of those examples. So this is it. The table names are completely integrated into Magento 2 as well. So now for a new eCommerce and junior developers, it's going to be very hard to understand what does super link mean. But for me as one of the former core developers who actually took part in creating this functionality itself,

, it's, it's quite obvious. What does it mean? So

Eric Hileman: [00:13:08] totally not obvious to me.

Ivan Chepurnyi: [00:13:12] So the super link is a table that is used to store relation of the configurable product, a simple product. And is there is also a table called super attribute and super attribute labels. Super attribute just tells which attributes are used to configure, these drop downs.

, on the front end. Let's say if you have color and size, you would have two records per configurable product in super link. Oh, so super attribute table. And there's also another one called super attribute label and the super attribute label is containing, a store specific translation of the dropdown name on the front end, and usually people don't enter any value, over there so it just, just over there and not used.

Eric Hileman: [00:14:05] Cool.

Let's let's, let's move on to the category page.

Ivan Chepurnyi: [00:14:09] Yeah, definitely. Let's, let's go there. So here we can, tell for sure just use collection and you can save a lot of time.

Eric Hileman: [00:14:19] Yes. That's a big win.

Ivan Chepurnyi: [00:14:21] Yeah. So which one he said was a problematic one uptown, right?

Eric Hileman: [00:14:29] Yeah.

Let me check.  

Ivan Chepurnyi: [00:14:32] so let's just run in profile of this category itself. Yeah.

I also need to put my microphone on a bigger stand.

Eric Hileman: [00:14:52] We're upgrading our equipment. My new camera is a little shaky too. I need a better stand as well. And you got a new light too. You got that nice light glow around you.

Ivan Chepurnyi: [00:15:03] Yeah. I got some professional, a lighting. , one of my friends who is a professional photographer, Keith told me after viewing the, we do a, Hey, you have to actually get some lighting because your face is dark.

Eric Hileman: [00:15:17] Yeah.

Lighting is important. It's a. I guess it's as important as your, your camera as well. Like even the best camera...

Ivan Chepurnyi: [00:15:27] And especially in my case the time when we record, it's usually late here in the Netherlands. So there is no daylight and I have only these standard office lights that is here and it doesn't provide enough lighting for video recording.

Um, but unfortunately I only set up for now only one light. , I have actually a set of two, but I don't have um, power outlets splitter in my office. , I forgot about it. So right now I can do it in such a way that is still lightens me up, but not as good as it would do with two devices being on sides around me.

Yeah, so if you have some dark places on my face, don't worry for the next time is going to be much more better.

Eric Hileman: [00:16:20] Mark Shust was just posting Twitter tweets about his new lighting setup. He posted some, he's got a new cam and he was adjusting his lighting and it's like everybody's getting into the audio video thing now.

We're all going to have little mini broadcast studios. It's addictive.

Ivan Chepurnyi: [00:16:38] Yeah. It's, it's, it's quite an interesting, you know, like we got so much a nice response from the audience. I was thinking like, Hey, we'll, we'll get a really complained about like, Hey, why, why do review our stores or something like that no, people actually much more interested in this kind of.

Yeah. A functionalities. And a lot of people actually now contact us to get reviewed on the show, right?

Eric Hileman: [00:17:04] Yeah. No shortage of content, even from our own customers.

Ivan Chepurnyi: [00:17:07] There's no shortage of content, and everyone likes it. So we have results here, right? So a 1090 database queries.

So it's already a thousand. Oh let me guess. Is it Anowave?

Eric Hileman: [00:17:35] Yeah. They did reach out to us and you gave them advice, um, were they receptive? Do you think they'll take it?

Ivan Chepurnyi: [00:17:45] Um, yeah, they were worried a little bit, about the adjective I was using during the last show about their extension.

Um, but I explained to them, you know, like if I have to remove this extension in every single merchant store. And because it introduced performance issues, you know, I can use any adjective I want however um, I realize the conversation we had over email with Anowave guys?

And they are really gonna take a look at what they can actually improve with the extension and they took seriously the advice I was giving about the product impressions. I still don't have time to explain more in detail what actually does they should do. Right? Yeah. Um, but, um, I hope that on one of the shows we finally will get that extension version that will tell that we cannot find any performance issues.

Eric Hileman: [00:18:43] Hey, I hope so. I hope they release it and we'll put it, you know, if they... our goal is to work with extension providers. So if you're an extension provider, you're working you watch this and you see something like, reach out to us. We want you to do better and we want to help you to improve Magento performance for everyone.

Ivan Chepurnyi: [00:19:02] There was a very nice case for me when one of actually extension providers improved their extensions based on my input. When I was working with a merchant.

Eric Hileman: [00:19:11] That was Nosto, right?

Ivan Chepurnyi: [00:19:13] Yeah. Nosto.

Eric Hileman: [00:19:15] Yeah.

Ivan Chepurnyi: [00:19:15] Those guys that are amazing. Yeah. It took me a while to get to the right person in a call.

Eric Hileman: [00:19:21] I talked to them at MLEU. Yeah. I talked to them at MLEU. And then they reached out and connected and they did. They published a new, a new release of it that fixes that issue. That's awesome.

Ivan Chepurnyi: [00:19:31] Yeah, and it's great because you know, on the merchant stores, I was trying to help with the performance to the merchant?

And also was introducing like couple of thousand of database queries on a product view page. So just because they were not dealing with the data that was available over there already but, you know, like then , they just changed the, according to my advice. And they just told me, told them just, Hey, here's a product object.

You already have all the data over there and they changed the code to use this as well as they improved the iteration of the product name breadcrumb. , for a, a, sorry, category name breadcrumb by using also the logic I suggested to them with the collection. As well. And now their new extension version doesn't have that number ofdatabase queries. It's fraction of times you know, fraction of times better.

Eric Hileman: [00:20:39] Yeah. They were, they were good to work with. So thank you, Nosto.

Ivan Chepurnyi: [00:20:42] Yeah. , it was really nice. Also, I was having a chat to do Magento Live EU with one of their technical leads and one of the technical leads really got into, um, you know listening and understanding what was actually a problem over there, because a lot of technical leads just, yeah. get the information, they they will pass it to the developer. But this guy he actually took it a all the way to developer and validated the complete implementation with the developer made sure that everything that I advised was implemented properly.

Eric Hileman: [00:21:24] Extension vendors get at us, let's, let's fix problems that we find and awesome. So what do we see here? That again, Anowave, which Anowave extension was it? It is the EC get impression push for, oh, is that the the tag manager again?

Ivan Chepurnyi: [00:21:43] Yes.

So it's the same tag manager and the same thing as before. Um, is there is no product, as you see? There is no products shown on this page? But they still load the whole product collection and access the message that triggered a load in a loop. Yeah. Going to close the homepage.

Here is a category page, and another one here is again a, I think just saying, a slider widget. So this is just security.

Eric Hileman: [00:22:25] Yeah. Okay.

Ivan Chepurnyi: [00:22:27] Yeah, it's, it's, I think trivial fix probably a developer can do it in  a couple of hours.

Eric Hileman: [00:22:34] Yeah. The developers on this site are pretty responsive, so I'm sure they'll get that fixed quickly.

Ivan Chepurnyi: [00:22:39] Yeah. Also...

Eric Hileman: [00:22:40] They've been good to work with.

Ivan Chepurnyi: [00:22:42] Here is something as well. I don't understand because yeah, here the biggest thing, but also as soon as let's say these two are gone, the next big big thing is going to be this one. Right? And what is here?

What is this? Okay,category get  json markup. and another and another one for us to take a look at is this time, it's another extension vendor called Mageworx.

Eric Hileman: [00:23:20] Okay. They were on episode two.

Ivan Chepurnyi: [00:23:26] , no. They weren't.

Eric Hileman: [00:23:28] There's something, one of the Mageworx... I'd have to look

Ivan Chepurnyi: [00:23:33] There are so many extension vendors that use Mage as a part of there company.

So it's very easy to there was a... I think I saw very similar issues before, I can't tell you for sure without taking a look at the code. But we can actually take a look at the, um, source code of the page itself. I have suspicions that, Oh yeah. And I already see it's so probably it provides, is this, um, open data format for a page.

Eric Hileman: [00:24:13] Oh, yeah. Okay. Maegworx was something we found internally. That was their option. Option inventory that we were having problems with. This is the SEO.

Ivan Chepurnyi: [00:24:28] Yeah. So was this, this is the open data format for you know, showing product offers, but, I can tell you for sure that this implementation is completely invalid.

, because if you take a look at the Google guidelines for open data and micro data on pages Google only takes one product per final page in order to be used in Google shopping. So if you have an open data that just lists a list of products. You can just throw it away entirely because it doesn't make any sense.

It makes sense to show product, a open data on the actual product page. But on the category page, it's completely redundant. It doesn't give any value. It actually can bring more issues because first of all, yeah you slow down the performance. Um, because here's they don't show any data.

And another thing is they actually get, can get banned by Google because they show product data in open data for products that are not shown on the page.

Eric Hileman: [00:25:52] Yeah. They're just showing the whole category whether those products are visible or not.

Ivan Chepurnyi: [00:26:00] Yeah. It's one thing and another thing open data in a specification.

So let me just open it. Open data. Um, Google search. Okay.

Microdata, all the time.

Structured data. So here how it works. So well, let's open right away. Um, the JSON LD specification and here in a documentation? Oh yeah. , let me find this single product specification. They don't list it over here. It's probably only Google, um, somewhere. I don't remember exactly.

Um, but there is some page where it tells us about different types of supported structural data formats.

Eric Hileman: [00:27:18] Yeah. So what's the, what's the marketing purpose behind this?

Ivan Chepurnyi: [00:27:23] , it's basically in search results to have any information about the product: what's the price, what's the availability with a picture...

Eric Hileman: [00:27:32] But you wouldn't want them to go to the category page though. You'd want them to go to the product page in a search result, I would think. I mean, I'm not a marketing expert, but So, yeah, I mean, it is redundant. Like you have the product pages and then you're putting all that in a category page.

Why? Why have both?

Ivan Chepurnyi: [00:27:55] Oh, it's schema.org Sorry. So schema.org. , is. This is the one with the data model. Sorry.

So let's go into the product. So here is schema.org. Any offer product or surface, for example, a pair of shoes a concert  ticket and so on. , and here is some things that is very important type things that you cannot have more than one product per page. I don't remember where exactly I saw it, but probably it's not schema.org, but it's somewhere on Google, Um, page here.

Eric Hileman: [00:28:58] Yeah. So what, what would you want a category page indexed as? , like the, the type? The, the type of items that are in it? Like if somebody is like, searching for, I don't know, sofas so, you would go to a category that shows sofas. So is that why they're trying to put like all, all the sofa products into that?

Ivan Chepurnyi: [00:29:28] Yeah. It just contains all the data and when you search, you only get one product: the data highlight. You're always a search result, and it doesn't make sense from point of view to include all of the products as open data. It just wastes your page size. You just waste your server resources.

, so you need only to show single product. If you have a page, just show it as page related to a category show, breadcrumb or something like that there's an open data.. But don't list all your products over there.

Eric Hileman: [00:30:04] Yeah, yeah. And that's what it's doing how much? 400 milliseconds, almost half a second to generate that.

Ivan Chepurnyi: [00:30:16] Yeah. and it just. It doesn't make any sense to do. So here. And it also adds a lot of data to the page. So recently I was doing some... Oh my goodness. Yeah. , so here is a, is this a data layer extension? , you see it has also a lot of inline javascripts.

so it's also not good for performance because first of all, every single script within the page is a part where browser is going to stop and , try to get a good script and only then it will continue rendering. So all of this JavaScript is, there needs to be Magento in it with a separate Javascript file being outside of Magento.

And this kind of inline scripts from extensions are bad thing for your front end performance, and it's very hard to actually to get, um, this JavaScript out? , because, okay. There's so many of it scattered around the templates of such extensions.

Eric Hileman: [00:31:48] Okay. At least it's at the bottom, but it does look like it's blocking on load.

Right. Or J query, asynchronous load. It's been so long since I've worked with jQuery.

Ivan Chepurnyi: [00:31:59] Yeah. So here you see script is after script and here is a require, so it requires some jQuery module. Yeah, it's going to be asynchronous , but still the time. , like for instance here, you cannot completely make your script to differed because of  this script...

Okay. So basically is it that it was it? So, um, what I was talking about is just don't inline your JavaScript in a page rendered. Always separate them. It's good from point of view of maintenance. And also it's good from point of view of the data.

Eric Hileman: [00:32:45] , okay. Do you want to take a quick look at the product page? I don't think we looked at that one.

Ivan Chepurnyi: [00:32:50] I think we wanted to look at the real category page. So for now we only were hitting the extensions that were working through the product that was not supposed to do it because there is no.

, products actually shown on the page. Okay, so now here what we have here, I'm just inside of another category and I think I need to go into one more category in order to see some products. Okay. It looks like some products are here, right?

Good. , now let's run some profile. so I'm going to disable aggregation again because it doesn't make sense. So probably there's going to be some performance issues because I see, our favorite  feature you know, on the category page...

Eric Hileman: [00:33:56] Swatches.

Ivan Chepurnyi: [00:33:57] Yeah. So swatches are always gonna show some nice issues.

And yeah, the rest of the things are I think gonna be good. So, yeah, 700 then the seventy three. And there is only like five products right. Oh, no. I think it sounds like..

Eric Hileman: [00:34:24] 20?

Ivan Chepurnyi: [00:34:27] We can check now in the timeline. how many products, actually we can do it here.

So thirty products or something like that,yeah. Thirty configurable products.

Eric Hileman: [00:34:51] So that's the, yeah, that's probably the product, um, factory, the full load in the loop.

Ivan Chepurnyi: [00:35:03] Yeah.

So first of all, yeah, we have this Anowave. Um, that again? Um, manages the data is pro... Oh, look.

Oh, this one? I think we didn't cover during the last, um, episode. Also, it's using something that is considered a bad thing? Um, it's using a product metadata class in order to obtain the version. But inside of Magento, this product metadata conversion is actually gonna hit file system by reading composer that Json from a file root of a project use composer classes in order to instantly eat it.

And it's going to be hard for memory, hard for CPU, and it's only through two of the Magento version itself. And I didn't know what this part of the code is used for. I don't know for what reason they need to product with a data version, because usually they use it for some kind of code pass differences. It's not a good thing because probably the data version doesn't tell you which modules are installed and which version those modules are for Magento core. The only way to restrict your code from, um, you know, cross, actually, how... Cross compatibility is the best way to do it is actually to define a dependency in your composer Json, on particular versions of the modules.

Um, you know, about issues with compatibility. It doesn't make sense to do checks. Like if product version is this, if it's enterprise, if it's a B2B, if it's a, um, you know, if cloud tools installed or something like that it won't give you real real code pass, um, differences, when you develop something.

So it's, it's always better not to do it. No. Not to use composer information or to put it inside of your. , extension code in order to verify is it gunna go this way or another way? So it's always a better practice in my opinion. Some extension vendors might disagree, but in my opinion it doesn't make sense.

Eric Hileman: [00:37:51] What extension was that in? Right here?

Ivan Chepurnyi: [00:37:54] The same. Google tech manager? I think.

Eric Hileman: [00:37:56] Ah. Oh, the Anowave. Okay. Yeah. The Anowave tag manager.

Ivan Chepurnyi: [00:38:01] Yeah. So it spends 200 milliseconds on this one, but it can be more with high traffic. So what I'm usually doing for my customers, I just create a module that creates a plugin around get version of product metadata and I just directly hard code string value of Magento installment over there in order to not fix this issue within an extension itself. But just using the product question doesn't make any sense. In my opinion in your production code, whether you are on the front end, it makes sense to access this information, let's say in common line or in a Cron job to do some license checks or whatever, but when you are on front-end page, when you're, um, doing something with products or doing sampling with templates and outputting some data.

You don't need Magento product question in your code.

Eric Hileman: [00:39:06] , why? Why are people using the Anowave tag manager? On Twitter, um, some people were saying like, yeah, Magento two has Google tag support built in. Some people said it's good. Some people said it's not so good.

Ivan Chepurnyi: [00:39:21] Yeah. I, I think people are using it in order to get some additional e-commerce tracking functionality.

Yeah, some additional functionality, because Google tag manager is all aboutthe data you push to it. So I think the standard Magento Google tag manager implementation doesn't have e-commerce data like shopping carts and so on. And it's very important to connect your Google tag manager with Google analytics and be able to track your revenue and so on and I think that's what a Anowave does. They provides this e-commerce data. As well as we know, that what people on Twitter said is that Magento provides this Google tag manager in their Magento commerce offering, not in open source.

Eric Hileman: [00:40:15] Oh, it's not in opensource. Okay.

Ivan Chepurnyi: [00:40:17] Yeah. So the extended tracking for Google tag manager is only available in Magento Commerce.

Eric Hileman: [00:40:24] Oh, that's a commerce only feature. I, I wasn't aware of that.

Ivan Chepurnyi: [00:40:28] Yeah. So that's why probably people would have to resort to buying extensions, but in general. You know, like it just makes sense maybe to invest for us as a community sometime and actually improve an extension of Jisse.

Eric Hileman: [00:40:49] Yeah.

He was asking like, should I improve mine and I, at that time I wasn't clear that it wasn't available and open source. And, um, Daniel had mentioned that he's been working on one, I think for a while now, but he didn't. He was like, just keeping his secrets and not sharing it. So I don't know if he's gonna make it a paid extension or how far along it is, but it sounds like there's a need for it.

Ivan Chepurnyi: [00:41:16] Maybe in general. I would love any, um, you know, extension that I can play around with without um, you know, like this kind of extensions, they still need some developer attention. Like you're still gonna... You are not going to create an extension that will work out of the box with your Magento theme. And how Anowave and other extension vendors are doing, they tried to avoid the problem of imcombatibility by actually grabbing data directly from the blocks that render products. But as we see, this is not a good solution because it provides performance issues because it loads products where they are not needed. It might introduce some unnecessary database calls to things that are not needed as well.

, so the best way, in my opinion, to, you know, work with impression tracking is actually build it all in JavaScript. Just provide for javascript, that configuration and tell, Hey, you can find product name this CSS selector. We can find product price by this CSS selector.

Eric Hileman: [00:42:30] The selectors.

Ivan Chepurnyi: [00:42:32] And that's all.

And then you can even build a feature that is going to be killing the market by using intersection observer and sends impressions only when customer actually sees this product. And then you can build much more better marketing campaigns, much more better. Um, you know, tracking...

Eric Hileman: [00:42:50] We just had on the last episode, Anowave was sending impressions data...

Ivan Chepurnyi: [00:42:53] It's a free advice for you. It's a free advice where you just build this skill or feature. It's better for performance and it's better for marketing as well. Yeah. So it's just crazy. Okay. , let, let's go. Um. Over here. So the product list. I think what we're going to see here, Oh, just too much here.

So category products and then we have the product list, and this is probably the standard Magento issues as I see... Oh wait a second. Not, not really. I don't see a standard swatches here. Mmm.

So what I see here get simple products. So yeah, basically it's another issue here. You see. Is this a specific project, specific module helper called simple products that's probably involved on every single product over there and then, it uses a repository tool or the configurable product again, and then it iterates probably over all of the simple product of that the configurable, you know, like actually is there a couple of load in the loops I think.

So basically I would recommend a developer to take a data by just creating some kind of after load observer on the product collection and just add data that you need by writing a query with product IDs as a condition in it. And just use this one single query instead of, you know, you sent the product repository grabbing the product by ID, um, because it triggers a lot of database operations, a lot of unnecessary objects.

Again, we are coming to the same issue over and over again. All the developers think in point of objects instead of thinking in point of view of the data that they need, right? So they just...

Eric Hileman: [00:45:23] Yeah.

Ivan Chepurnyi: [00:45:23] They just a think, Hey, what objects Magento provides for me?

Eric Hileman: [00:45:28] What property can I use this object? Okay, I got this property. I need to hit these other properties. So how do I link this object property to this object property, to this object property instead of just getting the whole collection from the database and one go, in one shot.

Ivan Chepurnyi: [00:45:43] And I must admit,  you know, like, it's not on, it's not, um, that obvious for developers.

Because Magento pushes forward this notion like you should use only our public API methods, and the developers are going to the extent like, Hey, okay, I have to use the public API methods, so I'm going to use the repository and in, and you know, Magento is introducing an issue by not providing a good way to add the data to the original collection of items and developers are just using, you know, that means that are available because Magento tells you to use their public API. No one talks about, Hey, but you can create on service model into which you supply IDs of products into which you can supply a store ID you are currently on and you can write bests for this service model and you can make sure that with the next Magento upgrades, your service model is still going to work by running test against future Magento versions instead of just relying on the Public API that any way going to be changed anyway might be broken in future so it doesn't make sense to lock yourself in into performance issues and into actual dependency on the classes that exist in the Magento in order to access the data because the classs that you are using right now might not exist in future Magento versions and Magento switches to graph QL. And it means like all of the code that is written right now by using product repositories is just going to go away. You know, like if you're going to use product repository, it means thatthe code that was working fine for you before, in future Magento version where they will completely remove repositories won't be, um, you know, available for you because they probably are gonna use for front-ends they are gonna use some graphQL end points. And. You better switch to that, but also in that case, I also don't want people to go on and start using graph QL endpoints.

No, it doesn't make sense as well.

Eric Hileman: [00:48:10] What does make sense then?

Ivan Chepurnyi: [00:48:12] Create your custom abstractions,

Eric Hileman: [00:48:15] Your service model, your own service model?

Ivan Chepurnyi: [00:48:17] Yeah. And is the internal implementation of the service model is completely in your control. And you can build a lot of code around this service model. So let's say you have a business logic that calculates price for all the products in the shopping cart? Right? And do you want to receive, um, those products, with the prices from the standard Magento, let's say SKU and price and do you want to run this SKU and price where, through some business logic you have. And let's imagine, like if your code depends on the repositories, it means that every single time Magento changes its API, you have to write the business logic itself. So what usually developers do is they create an abstraction, that will return only data they need , let's say SKU and price as a basic, simple types.

Not as the whole product model, just the data they need to perform this business logic and how this data comes in, should not be, um, should not be written within a business logicthat calculates the price. The price just knows, Hey, I received an array of sku and price pairs and I just returned back some kind of result of my calculation.

I don't care how I get these pairs. So this as a basic idea of why you should create service models and your own abstractions. You can even write your own abstractions that will still use the repositories behind the scenes. Like you don't need to work directly with the database. We can still use repositories.

But as soon as you have this layer of indirection, it's going to be easier for you to maintain upgrades from version to version. , from a upgrade to upgrade without rewriting a big piece of code. You have a very easy  couple of lines that you need to change if Magento changes something.

You don't need to change these lines of code everywhere around your code base.

Eric Hileman: [00:50:25] And if you're an agency, you could reuse that then from project to project.

Ivan Chepurnyi: [00:50:29] Yes. So in general, always create your own abstractions. It doesn't matter. Like, yeah, it might take some extra effort to create some additional class and maintain it, but it's worth the effort because all of the time you're going to have changes in some third party like Magento or any other extension as well as they might introduce changes as well.

And if you directly use their code with your code it's going to be a problem for you because the more places, where the code can change. the more places it will fail in the future. And I was doing a lot of talks about the challenges of architecting Magento customizations in Magento two.

, I was giving a talk I think I started doing it in 2015 and I have some slides and, um. Also video for my talks on Magento conferences where I was speaking about exactly this problem: that you have to create abstractions around Magento core, around other extensions in your code in order to be able to keep pace with the changes in other systems without impeding your development process?

Eric Hileman: [00:51:55] Are they still online? We can link those in the show notes?

Ivan Chepurnyi: [00:51:57] Yes. We'll definitely link those in the show notes.

Eric Hileman: [00:52:01] Okay.

Ivan Chepurnyi: [00:52:01] So it's not only about performance, it's also about maintainability also about, um, you know, making your life easier as a developer.

Eric Hileman: [00:52:11] Yeah. Um, can Blackfire profile adding a product to the cart.

Ivan Chepurnyi: [00:52:18] Yes. Oh wait, so here, here we see it's the same thing. You know, like just Load in a loop. Yes. Remove this load in the loop. And everything is going to be fine. So we probably have first to go to the, um, product page. Right. So let's just profile the product page. Yeah.

So, first of all, when we are profiling, when we are just on the page, we can run a profile button. But shopping cart is usually an operation that involves a couple of steps, you know, like it's going to ajax request and you cannot profile ajax request without, you know, just with a push of a button because you don't have a button , is that willprofile specifically this specific request. Okay, so product page. Let's just quickly look at the timeline. Maybe there is something obvious we can fix or suggest to fix. Oh, let's see. Okay. Mmm. that's quite a lot to render attributes.

Attribute repository.

Okay. There is another load in the loop. , so a developer needs to look into this help post attribute, get product attribute because he loads separate with product, again, on the product page when he already has a product. So basically just fixing this can help you a lot. Because this is basically loads of problems if it's already a loaded and it's completely redundant.

Eric Hileman: [00:54:27] That should be easy then. Okay.

Ivan Chepurnyi: [00:54:31] So easy fix just remove this load and access your current product that is already available on the page. No need to run a separate query. Okay. Let's go back to, um, a thing called add into the shopping cart.

Right. So in, in blackfire, unfortunately, I think viewers won't see it in my recording. Um, by when you open in Blackfire, you have a one big button called profile and is there is also another thing called profile all requests. So if you want to profile add into the shopping cart, you need to click on profile all requests and you need to confirm it by clicking the record.

And then it will show a small window with all of the requests that are happening. And that has been profiled. 

Eric Hileman: [00:55:21] Only your requests?

Ivan Chepurnyi: [00:55:27] . Yeah. Only your requests of your browser.

Eric Hileman: [00:55:33] Oh, okay. Okay.

Ivan Chepurnyi: [00:55:35] So request of the browser that hit PHP?

So now I'm just going to add this product to the cart and , I already see that there is at least,  three requests are happening. So one of them is adding to the cart is that this a post and two others are related to the customer section load. So let me stop the profiling session and I am getting directed right now to the page where requests are profiled. So checkout cart app. So this is a request is it this executed during adding to cart?

Let's take a look at a timeline. And I think there's nothing strange happening just a standard add to cart, um, by using API, um, go back to the store. Those, yeah. Pure core Magento nothing, nothing strange happening here. If this customer would have MSI enabled, we would see a lot of database queries here.

Eric Hileman: [00:56:42] Oh, yeah. MSI. Well, why was it two requests to the get customer cart?

Ivan Chepurnyi: [00:56:53] I think it's an issue with Java script. Um, because...

Eric Hileman: [00:57:00] Are they the same?

Ivan Chepurnyi: [00:57:01] Yeah. , I think in 2 point... 2.3. there is I an issue when Magento does, Oh, look, because there's two requests.

Eric Hileman: [00:57:16] Ahhh,

Ivan Chepurnyi: [00:57:17] To shopping cart? Is there a session, a code getting triggered? So it makes sense for a developer actually to disable a reddis session locks. yeah. Yeah. And the standard Magento, a customer section load, it just loads the information about the shopping cart.

Eric Hileman: [00:57:42] Sure. Well, was there a bug that it loads that it makes two calls?

Ivan Chepurnyi: [00:57:47] Yes. There is a bug and I think there is already some fixes for it, butthe whole concept of customer section data is wrong because, um. When you are doing full page cache, you want to reduce as much as possible requests. So your Magento backend and when you open any Magento 2 store in production right now, if you open a new Relic and you'll open transactions and you'll sort the transactions by the most number of requests, the most number of requests are always customer section load because Magento, just on every single page triggers Magento backend anyway, so it doesn't matter are you behind Varnish or not Magento job decides, Hey, I need some dynamic data to retrieve from a server. However, when you implement a project, it's a good practice to create a guards on those http requests. You want to make sure that they will only way how Magentotriggers a request is one customer actually performed some dynamic action.

So if you are an anonymous user and you're just browsing the website, you don't need to check your shopping cart because you didn't add anything to the shopping cart yet, right? You are not logged in.You didn't log into your account.

Why we are calling customer section load in those cases? It doesn't make sense.

It just wastes resources on a server. It triggers unnecessary session starts in the Reddis. Um, it consumes valuable server resources. So what I usually do for customers say, i  just depending on the customer implementation, of course, because I cannot tell if it's a solution that will fit everyone.

, but in general. I just create a JavaScript customization is that, um, has some cookie guard. And this cookie guard is a solution where I just specify, um, Magento rules in the backend based on which I will set a specific cookie value let's say is a timestamp. And this cookie value with a timestamp will have,  a meaning for your JavaScript. So JavaScript will read this cookie, and if it cannot find this cookie at all, it will prohibit any ajax request being called for customer section data. It will directly assume that,there is an empty shopping cart, customer is not logged in, there is nothing happening. Just return an empty Json result as a place holder.

Eric Hileman: [01:00:35] Stay off the backend.

Ivan Chepurnyi: [01:00:37] Yes, stay off the backend and um, if the cookie is present. What it will do, it will do an ajax request with the needed data to be loaded from the backend, and it will use this cookie value as an indication for future requests. So what it will do. It will take this value. It will take the data received from the backend and it will store it all in customer browser's session storage. So on the next vist of the page, it will check, Hey, what does the current cookie value, what does the cookie value that I've stored in my local storage, are those values the same. If they are the same, I just take the data from the local storage of the browser. I don't do http request. And in case the values differ, then, yeah. Okay. I need to do an http request to get this dynamic data from backend because something has changed. Customer did some dynamic operations that affects the state. Also, dynamic data in a browser.

And in such such a way. Those customers actuallly load.... Um, no, I'm going to work on something open source around that.

I actually, have this solution for a long time. I was even talking with Magento core engineers in 2013 when Magento two was still in very early, early, early stages, and I was sharing them with an implementation I was doing for my ecomdev varnish module for Magenta one. Where I had this solution being implemented from the day one of the module. Because the most important thing when you use in Varnish, you want to make sure that you don't hit backend at all unless you really need it.

Eric Hileman: [01:02:33] So public service announcement real quick. Make sure your email campaigns or your marketing campaigns, any variables that are appending onto the request are accounted for in Varnish, or you're going to blow right through it. We've, we've had quite a few people do that recently.

Ivan Chepurnyi: [01:02:48] Yeah, it's, it's basically, I think we were talking about a couple of shows ago. We were talking about that peo ple just don't modify VCLs that are provided by default with Magento, you need to modify this VCL you need to adjust it to your marketing tools. You need to verify it before, um, you are marketing campaigns on a have a sale season, like Black Friday, Cyber Monday, and Christmas sales. Make sure that you verify everything before you go and get your servers hit with swarm of buyers who want to go and buy products from your store but they cannot because everyone receives a unique version of the page that is uncached by Varnish. And everyone just gets frustrated and leaves your store without any purchase. So make sure that you configure your stores properly. So, yeah. Okay.

Eric Hileman: [01:03:44] That was good. So you got the slider in the beginning. , that's going back to the product repository. , the, the tier price that's already in a, in a collection and the plugin for the price box, and then Anowave tag manager. And once again, hopefully the discussions with Anowave go well and we can get some improvements with those guys. , the Mageworx SEO markup, best to remove that, that could result in bans right from the duplicate products being indexed.

Ivan Chepurnyi: [01:04:21] Um, it's basically, it makes sense to have open data only on product page. So only on the product should be on the product page.

Eric Hileman: [01:04:32] Cool.

Ivan Chepurnyi: [01:04:33] Let me see. First of all, you, you want to hit your, you want your customers being um, directed to the product page, not to category page because you can change the position of the products every single day. The product may go on top on the bottom of category listing very often. And. , Google is not updating that often category pages in its own index comparing to the product pages that are quite stable. Product has own URL and it's always reachable by it. So, yeah, it's just useless to output open data on the category page. , I don't know. Um, maybe, . Someone will put a comment in this show and tell, Hey, I'm an SEO expert and I know that this data is very valuable. I'd like to have an open discussion about it because I don't see any reason to...

Eric Hileman: [01:05:36] I'll tweet it out. Or you tweet it out, and I'll retweet it, whatever, whichever. We can get a discussion going around that, that'd be great.

Ivan Chepurnyi: [01:05:45] Yeah, well it would be because a I see from time to time, a lot of SEO based extensions, they just implement something without understanding the topic of search engine optimizations and people just in case this extension solves a problem. But usually you need to really get some expert to look into it. And. I, I don't trust a, you know all the SEO Suite modules from extension vendors because of that, because I was burned so, so many times already. I was reviewing so many merchants in Magento one stores, in Magento two stores, that have those kind of extensions installed. And. It usually just makes sense to talk with your developer and check which things actually need to adjust in your theme in order to get better SEO scores. Then it just means that, Hey, it's selling this extension.

Eric Hileman: [01:06:45] Yeah. Just dropping an extension and I'll magically be on the first page. No.

Ivan Chepurnyi: [01:06:52] Yeah. You need to be very careful with these things.

Eric Hileman: [01:06:59] The simple products? We saw that one where, . Um, was it the, was it the swatches or was it the category page where it was, um...

Ivan Chepurnyi: [01:07:08] It was custom helper. Build custom helper for this project.

Eric Hileman: [01:07:14] Yeah, I have that on the list too, but there was one before that, I think it was the regular product repository load on SIM, privates.

Ivan Chepurnyi: [01:07:23] It's exactly something that is apart of project customizations here.

Eric Hileman: [01:07:30] And then there's two calls to the customer get, get cart. That's a bug you said? That's fixed in newer versions of Magento 2?

Ivan Chepurnyi: [01:07:39] Yeah, I think they reduced it to one, but as I said, it doesn't make sense to do any calls.

Eric Hileman: [01:07:45] Yeah, yeah. Well, until you can release a module for everybody, I think...

Ivan Chepurnyi: [01:07:52] When I release something, it's still not gonna be something that, Hey, I can install and it will magically work.

Eric Hileman: [01:08:01] Everybody can, out of the box use it. Yeah.

Ivan Chepurnyi: [01:08:03] It won't happen likw that. You still have to adjust it to your needs because I can do it for one case. But you know, some people have so many different customer sections. Maybe they introduce a custom data being added to this customer sections, they need to add. It will be extensible. So you would be able to do it for their own modules, but you still have to do it as a developer. It won't just work magically. You need to tell which sections to load based on which customer actions.\

Eric Hileman: [01:08:40] Cool.

And then there was, there was one more, I think it was just, let me check my notes. Oh, the Reddis. Reddis session locks. Don't use... I mean, is that good general advice for everybody? Don't use reddis session locks at all.

Ivan Chepurnyi: [01:09:01] I was recently reviewing the code. I'm like, it's possible that sometimes it would happen that the data gets corrupted, but I was thinking it's possible, but I was actually reviewing, the code and , he actually wrote a very nice adopter for sessions, and it's already using pipelines of reddis in order to store the data. So the data is actually stored automatically. It's not stored as, you know you've write one value then the other value, it always happens automatically. Yeah. Sorry. Right.

Eric Hileman: [01:09:42] Reddis is single threaded, I thought it only uses one CPU, like it's not, maybe that's, yeah, I thought, I thought like you can't do that and you can't override stuff in reddis the way it's implemented. Cause I, I know it only uses one CPU, but under the hood like do they have some sort of multithreading model under there that could actually....?

Ivan Chepurnyi: [01:10:05] No, it's, it's single threaded from point of view of data storage. But the thing is with reddis. It's, it's not something that you write to file because when you write to file in PHP, if you open, if multiple connections start to open connection to open file, there is really random chance of some data being out of order and it's just terrible. So that's why no one using file based sessions in production. Because if you have too much connections, even if you are on a single server. Now filed based sessions are gonna really impede your performance. And as well in file based sessions, it does make sense to have a look, because the operation of writing to file is not atomic because everyone can open a handle can write some bytes, and those bytes will be decided by operation, by operating system, after which a process will write into which  position of the file, but it can be a weird situation where your file just gets corrupted. So that's why PHP automatically locks, writes on sessions in file mode. But in reddis it doesn't make sense because every single request to change the data in reddis is an atomic operation. So it means the last process who wrote a reddis session will be the winner in with the data. Um, but it can be a situation like let's say if you want to have only one process writes at a time because you, you want to make sure that the data that is always written once and there is no situation, like in one connection, customer added, customer logged out, but another one it logged in and then, it was at the same time. And then you have, um, someone who, who, who wrote the last data and for instance, if they, there was customer being logged in, has been written after customer has been logged out it means customers is still logged in from point of view of a session. Right. So it can be a problem, but in general in Magento those problems. Mmm, do not exist. Because the session is not something that contains really Zillow valuable data that it gets used later on. It just contains information, what, what does your customer ID, um, what does your code ID? And so on. And basically here

Eric Hileman: [01:13:00] It was Reddit sessions as well. So we do split that across three. So reddis sessions are in their own reddis process and config cashes in another, and then page caches and another. Oh, you have isolation there as well. All right, well let's get the, let's get the conversation started on Twitter. About. Using, um, the product data on the category page and see if there's any better or other alternatives to the Google tag manager extension and hopefully Anowave keeps communicating with us and gets some improvements made to their, to their extension, since we seem to be seeing a lot of people using it.

Ivan Chepurnyi: [01:13:42] Yeah. You know what I'm worried about is that I get to another set of emails because we mentioned, we are mentioning too much the name of the extension vendor.

Eric Hileman: [01:13:54] Hey look man, like the truth is people complain a lot about Magento 2  performance and yeah, it's, there're some things that could be done more performant and the core, but to be honest, as a hosting provider, a lot of the times. Just about every time, to be honest, when a customer complains that they're having performance issues, it's always coming down to third party extensions. So if we can find the popular ones, work with them and improve them, that's gonna improve the performance for Magento to overall a great bit...

Ivan Chepurnyi: [01:14:29] Yeah. Extensions. Right now, I struggle a lot with the master shop by extension. This is,

yeah.

This is a very interesting one. They have some piece of a code in one of the helpers, um, which grabs all of category IDs in the store. So if you have a couple of thousands is gonna take all of those category IDs and then it builds a huge, terrible query that will grab all of those category IDs. Um, merge them together in order to receive all the URL rewrites of all the categories by those IDs. And then it also does an iterration in PHP. And when you have a reasonably large number of categories in your store, this process can take one second. And is this piece of code is happening in constructor of the helper that is not all the time used. It's one thing and another thing, it happens on every single category and product page, so, it's like it's one second just iterating through all of these huge results that.... And yeah, I, I wish, I wish we had this show earlier when all those extensions were just starting to be developed, so we could educate developers better on the way how to do their job better to make merchants happier because no one is against extensions. You know, like I'm, I'm completely in support of extension market as soon as people care about is their craft. Extension developer network, I think is a good case of people who are really worried about extension of quality from point a few off maintenance from point of view from point of view of guidelines. Yeah. Unfortunately, not a lot of people pay attention to performance as a metric, but I think from point of view of the merchant. It's one was the most important ones, because you know, no one cares about the code style if extension performs very badly.

Eric Hileman: [01:16:42] Yeah. Yeah. It seems like a lot of these extensions came to market quickly. Well, came to market quickly in a sense that they needed to get them written and they needed to get them released. But I really, I think the problem was a chicken and egg problem where there wasn't a whole lot of market share for Magento 2. But. The extensions providers, they needed to write the extensions in order for the market share to gain traction because there wasn't a lot of extensions. So I think a lot of them got created with not a whole lot of knowledge in Magenta 2, and kind of in a limited resources way. But now that magenta two adoption is growing, hopefully it does make sense financially for extension vendors to invest more money into improving these performance issues and improving their extensions, that'll help Magento 2 adoption and they'll sell more extensions.

Ivan Chepurnyi: [01:17:37] Um, but let's be clear, I think there're development power houses. You know, they have huge teams of developers working on it. Um, as well. We have a huge Magento community ecosystem with a lot of developers, just general knowledge problems that people, um, just stop thinking in terms of you have your resource usage, stop thinking in terms of your database operations. Stop thinking in terms of view. , you know, the most efficient iterations for the data structure because. It doesn't show itself up on the small data or a small piece of code. And with Magenta 2. It's much more...

Eric Hileman: [01:18:25] You think that's what it is? Do you think, like.. They run it on their... Well, first of all, do you think it's a, a lack of knowledge and understanding of server and hardware and how those services are utilized?

Ivan Chepurnyi: [01:18:37] I don't think that's the problem exactly. I think I know a lot of developers who have a good understanding of the server hardware and so, and then they still do the same mistakes. , that I tried to prevent. But the problem in my opinion, is not in that area of the knowledge. The problem is in the focus. How have you developed? Because right now you have very narrow focus of your attention. So you work on very small class in hundreds of other classes. You don't see the whole picture. You don't see how the data gets transferred from one class to another class all over as a chain and you just completely lost all of the relation to the database to the way how Magento actually load its data you are using. Yeah. So that, that's why I always stop thinking in terms of objects thinking in terms of service models that will return you everything up front in one. 

Eric Hileman: [01:19:47] Okay. Okay.

Ivan Chepurnyi: [01:19:48] So if you iterate over a list of products, do not do operation on every product, try to collect all the product IDs, get all the data you need by those product IDs, work with the data that you preloaded for those products itself. Do not try to access repository or anything. Who was in this situation? Always think from point of view of you already have a data when you're doing something with this data not from point of view, like I need to get this data from somewhere because as soon as you start to think, I need to get this data from somewhere in order to do this thing, you're already doing it wrong because you were supposed to get this data upfront before you start to do something with it.

Eric Hileman: [01:20:33] Do you know?

Do you know? Is the Magento extension marketplace, Magento marketplace. Do you know if they do any performance testing on these extensions?

Ivan Chepurnyi: [01:20:44] , I actually was talking with Magento core team about it. , I think couple of years ago we were talking about the marketplace quality program. And so, um, and they were thinking about doing performance testing. But with performance testing you never know what is a configuration of an extension needs to be in order to trigger a performance bottleneck. You can have a million of products, but let's say the table on which extension realize the trigger performance issue is a table of the extension itself. And if this table is empty. It will be just never shown on performance profiles.

Eric Hileman: [01:21:25] Yeah, we get that too. Like people say, I'm having performance issues, and we'll say, okay, did you change any code lately? And they'll say, no. Then we'll go in and we'll find a query that's performing terribly and we'll say, well, it looks like it's this query, and they'll say, well, that query has been there for a long time, and we say, yeah, but the data hasn't. The data has changed and now the changes in the data are causing this to now cause a problem, and they say, Oh yeah, you know what? We did add some attributes, or we, you know, or maybe they just have a lot more data, a lot more rows in the table that it's now working on and it's struggling or, so, yeah, it's, it's, it's a test for that kind of stuff would be difficult, but maybe they could use, maybe they could look for load in the loop patterns by using Blackfire.

Ivan Chepurnyi: [01:22:13] Blackfire is easy.

Eric Hileman: [01:22:14] Count Product repository. Count the cause of product repository...

Ivan Chepurnyi: [01:22:19] Like any, any, any kind of profiler, you know, works well. So either  Blackfire. Either it's a native Magento profiler. Native Magento profiler doesn't show you much of information, but it can tell you which queries has been executed. So my general rule, if you have a page with a performance more than 60 database queries, you have a problem.

Eric Hileman: [01:22:46] Oh yeah. That's a good metric. Number of database queries.

Ivan Chepurnyi: [01:22:49] Yeah. Number of database queries per page is a good indication either you're doing something wrong or not. So as soon as you have the number above 60 you know, it's, it's already an indication something is going on. Because I usually have a rule. I have one database query per type of resource I want to access. So if I want to access a product. Yeah. Product has, let's say, five different kinds of resources, you know, like the different tables that I want to access data from. Yeah, we'll have five queries for list of products, but I should never consider that I execute the database query for every single product separately for all of those five different resources I want to receive for a product

Eric Hileman: [01:23:38] Do developers even. Yeah. Developers probably don't even think that they probably just like, Oh, I'm just calling product repository and I, this is magical and I should already have this. So they're not really understanding that they're going to the database.

Ivan Chepurnyi: [01:23:52] , things if you'd like to look at, um, some of the talks from Uncle Bob. Oh, Robert C. Martin.

,He is talking about, um, the failure of frameworks and ORMs. , I, I, I will try to get out and talk when he talks about this specific problem, when people just, . You know, use an ORM in order to access the data.

Eric Hileman: [01:24:28] Yeah.

Ivan Chepurnyi: [01:24:29] In the places where they actually don't need ORM at all.

Eric Hileman: [01:24:33] Yeah. And from the server side, the database queries that we see that come out of using objects to build database queries are horrendous. They're just the worst things we've ever seen. You see two megabyte, five megabyte queries that just return a single ID, like, Oh my gosh.

Ivan Chepurnyi: [01:24:51] Yeah, I think during the training with your team I was speaking about,  finding a balance between number of IO operations and the complexity of your IO operations. So you can reduce number of IO operations by increasing complexity, but you need to find the balance. Sometimes it makes sense to execute 10 queries rather than one query that is huge and contains a lot of joins. But let's say 10 queries by resource type is also good because you only access specific data storage. And that's all. By specific identifier so it's much more faster for MySQL than trying to join all of these ten separate tables together. Then Magento... then, MySQL actually multiplies all those rows together because joining is always multiplication by keys. So, yeah.

Eric Hileman: [01:25:48] Well, I did, I did a terrible job keeping us moving along and on track for time. We're at like an hour and 30 minutes on my recording, so we should probably wrap this up,

litl@magemojo.com. If you'd like us to review your store we'll link Ivan's and my contact information and Twitter handles in the show notes. Anything you want to add before we wrap this one up?

Ivan Chepurnyi: [01:26:14] Yeah. Developers use profilers. Use Blackfire

Eric Hileman: [01:26:20] Extension developers. Use profilers,

Ivan Chepurnyi: [01:26:23] Project developers as well. Solution partners use profilers. Magento Core Developers use profilers too.

Eric Hileman: [01:26:35] Yeah, there you go. Okay. All right. Well thanks everybody who's watching up until this point and we'll see you next time.

Ivan Chepurnyi: [01:26:43] Yeah. See you next time.

Leave a Reply
  • Cisco
  • Intel
  • Redis
  • Magento
  • Nginx
  • Dell
  • Percona
  • MemCached
  • PCI Compliant
  • BBB