I still remember. Around two weeks ago, when I was sitting on the very couch I am writing this blog, I thought I was super smart. Learned a new technology and used it to prevent spam. To save my customers from fake signups to their Ghost site.
Well, I was super smart. And sabotaged myself along the way. Welcome to the tale of how I just spent 6 hours debugging something incredibly stupid.
Yes, this will be a bit technical. But last time I wrote a technical post, a good friend (she calls herself a tech dino) said that she had no idea what I was talking about, but she enjoyed it. So uhhh...here's a techy nerd post to relax and laugh? 🙃
Magic Pages uses an Nginx ingress controller with ModSecurity rules to protect customer websites. (I did promise you a techy post).
Or simply...every request that gets sent to a customer website must get past a bouncer that has a list of rules that cannot be broken.
And one of these rules was not to use known SMS gateway domains in the body of the request. Why? Well...as of a few months ago, Ghost sites all over the world are target of spam attacks, where someone signs up hundreds of SMS gateways to people's newsletters. More here:
ajfriesen
The rule was designed in addition to Ghost's own protection measures, since there were more attack vectors after the initial wave. So, I wanted to be in full control.
And hey, the rule worked flawlessly! Too flawlessly, in fact.
Around a week ago, a customer reached out and told me that he cannot save a configuration in Magic Pages' customer portal. That's basically a unique part to Magic Pages. You can change certain parts of your Ghost site, which is usually reserved to self-hosted sites – not fully managed ones like Magic Pages.
At first, I thought that it must be a validation error. Something in one of the recent updates I rolled out. But the more I looked into it, the less likely that became. A 403 error usually means that the user is doing something that isn't allowed.
However, I was too focussed on authentication, and forgot about the most obvious...
So, today, I sat down and started an hour long debugging session. Set up test environments, deployed test versions, went through hundreds of lines of logs. Nothing. I could not find anything.
My wife and I then went outside for a walk. Had some ice cream, enjoyed the sun. And when we got back, I sat down again. Played everything through.
Then I asked Claude 3.7 to go through logs with me. A second (artificial) brain (not really, but let's not get into details...) is always good. But Claude was as clueless as I was and just suggested garbage.
I don't know how I got there, but eventually I thought...let's look at the access logs of NGINX. You remember...the bouncer.
And there it was. Light as day.
SMS gateway domain detected
SMS gateway domain detected
SMS gateway domain detected
SMS gateway domain detected
SMS gateway domain detected
SMS gateway domain detected
Seriously?
I have blocked my own customer portal. I nearly started laughing.
Indeed, the security rule was correct. Saving the configuration was a request that had a body with these known spam domains in there. Why? Because the configuration includes the option to set additional domains that should be blocked:
So, every request to save the configuration, included these spam domains, since they were sent to my backend server.
And NGINX said nope. Blocked them.
Well, good job, NGINX 😂
Next time you wonder why something isn't working, take a step back. Get some ice cream. Enjoy the sun. Then think about me. And how I sabotaged myself.
And if you're a Magic Pages customer who tried to save a configuration in the last two weeks – I'm sorry. You weren't doing anything wrong 😭
Now if you'll excuse me, I need to go make sure I haven't accidentally blocked myself from publishing this post.