In some cases, a change can go unnoticed

Hey there,

First off, thanks a ton for bringing this issue to our attention. Your detailed explanation helps us understand the situation better, and we appreciate the time you took to write it down.

Now, let’s dive into the core issue. We’ve given this behavior some thought. The way we see it, alerts are typically indicative of defects or changes that need to be addressed. So, when an alert is resolved, we take a new reference screenshot to ensure that the “new normal” is indeed what you intend it to be.

I get where you’re coming from with your proposed solution, but if we were to use the approved alert as the new reference screenshot, it could create a loop of sorts. Imagine you approve an alert that contains a defect. If that approved screenshot becomes the new reference, you might get another alert when the defect is actually resolved, which could be confusing and counterproductive.

That said, we’re always open to refining our processes. Your feedback is invaluable, and we’ll definitely keep it in mind for future updates. If you have any more thoughts or questions, feel free to share!

Cheers
Steffen

Thanks @steffenbew for your detailed response.

The way I see it, an extra alert is much better than an unnoticed change.

In our case, it took us 4 days to notice, by chance, the disappearing price, and it probably cost us some sales.

Isn’t that exactly what VRT are for?

Another solution might be comparing the new screenshot to the approved alert, and notify if they are different.

Think about it this way: in our case, the new screenshot reference ISN’T the one we approved. Doesn’t it miss the whole point of the test?

You’re absolutely right @asafm7, the point of having such a system in place is to give you peace of mind, knowing that any unintended changes will be flagged.

At the same time, it’s crucial that the alerts we send out are actually meaningful, so they’re taken seriously and prompt a quick response from the recipient.

I totally get your frustration about the disappearing price and the potential lost sales. That’s a big deal, and it’s something we take seriously. The idea of comparing the new screenshot to the approved alert is a solid one. It’s like having a second layer of checks to ensure that what’s live is what you actually approved.

We’ll definitely look into implementing a more robust process that can catch these discrepancies.

Thanks again for bringing this to our attention. Your input is invaluable for making VRTs better.

Hey @asafm7, I was just thinking if it could make sense to not pause tests and require alert approval, but to always take the latest screenshot as the new reference and continuously compare them.

Alerts would then report every detected?change from one screenshot to another. They would simply serve as a reporting feature and could be dismissed.

That way, every change could be spotted. However, it could be less obvious what the final “new normal” state is, after having dismissed a couple of alerts in a session.

What’s your take on this?

@steffenbew that’s an interesting idea to think of in general. It might solve or improve other aspects (an alert can potentially be solved automatically). But I don’t believe it will solve the case we are discussing, as tests will never be in real time. There will always be an interval between the test and the approval, and in this interval more changes can appear and go unnoticed.

I think the main point here is that the approved alert should represent the current state of the website, or be set as the new screenshot.

I’m not sure yet why the first idea of setting the approved alert as the new screenshot isn’t the best. If it doesn’t represent the current state, a new alert will be triggered the next day, and that’s make perfect sense. I didn’t follow the argument about a possible loop, but maybe I’m missing something. It can also save the computing resources of creating the new screenshot, and in general make less room for issues.

I think it also makes a semantic sense: the alert is approved AS the new reference.

• This reply was modified 1 year, 2 months ago by asafm7.

@asafm7 I think what I proposed would be equivalent to your proposal, but with the addition that tests aren’t paused. This could have the benefit that if even while a previous alert isn’t resolved, an additional alert could trigger for a new detected change.

Currently, the semantics of the alert button is to “resolve” the change, not to “approve” it. Our thinking is that a visual regression typically represents an unwanted change. That’s why we aim to prevent raising another alert a day later saying “all good again” for every defect that was detected.

I see how it might be less confusing if that “confirmation-alert” would be raised right after a resolve/approve/dismiss action. Please give us some time to consider all options to make the best decision moving forward.

• This reply was modified 1 year, 2 months ago by steffenbew. Reason: typo

Thanks @steffenbew.

My only additional input currently is that in dynamic websites, such as e-commerce, alerts are more common and usually are false-positives. Excluding elements can help in some cases, but not all.