Bad htaccess rewrites – causes 404 errors on all subpages

  • Several of our highly visited sites have had issues where we’d get 404 errors on subpages. This usually means the htaccess file has been corrupted so resaving the permalinks page is the quick fix.

    It appears the HackRepair blacklist is being updated at random intervals and not including the basic WP redirects when writing back to the file. I’ve copy/pasted the most recent htaccess file that was broken early this morning and you’ll see the WP lines are gone.

    The client always finds this before I do and his patience is wearing thin.

    We’re using Litespeed on our server and this plugin on well over 50 sites. This seems to be happening just on our more highly visited sites.

    Also, the first four lines were repeated about 150 times; I removed them to shorten the topic but I do think it’s important. I just verified and every site that we use the HackRepair blacklist, this is the case. I checked with 5 sites not using HackRepair and the 5 sites that do and the lines are repeated on only those with it enabled.

    Is there a chance as the htaccess file gets inflated with these repeated lines that the file has issues being rewritten by your plugin?

    —————————-

    # BEGIN iThemes Security – Do not modify or remove this line
    # iThemes Security Config Details: 2
    # Quick ban IP. Will be updated on next formal rules save.
    # END iThemes Security – Do not modify or remove this line

    # BEGIN iThemes Security – Do not modify or remove this line
    # iThemes Security Config Details: 2
    # Quick ban IP. Will be updated on next formal rules save.
    # END iThemes Security – Do not modify or remove this line

    ********* ABOVE FOUR LINES REPEATED ABOUT 150 TIMES ******************

    # BEGIN iThemes Security – Do not modify or remove this line
    # iThemes Security Config Details: 2
    # Quick ban IP. Will be updated on next formal rules save.
    # END iThemes Security – Do not modify or remove this line

    # BEGIN iThemes Security – Do not modify or remove this line
    # iThemes Security Config Details: 2
    # Enable HackRepair.com’s blacklist feature – Security > Settings > Banned Users > Default Blacklist
    # Start HackRepair.com Blacklist
    RewriteEngine on
    # Start Abuse Agent Blocking
    RewriteCond %{HTTP_USER_AGENT} “^Mozilla.*Indy” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^Mozilla.*NEWT” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^$” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^Maxthon$” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^SeaMonkey$” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^Acunetix” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^binlar” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^BlackWidow” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^Bolt 0” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^BOT for JCE” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^Bot mailto\:craftbot@yahoo\.com” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^casper” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^checkprivacy” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^ChinaClaw” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^clshttp” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^cmsworldmap” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^Custo” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^Default Browser 0” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^diavol” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^DIIbot” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^DISCo” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^dotbot” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^Download Demon” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^eCatch” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^EirGrabber” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^EmailCollector” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^EmailSiphon” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^EmailWolf” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^Express WebPictures” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^extract” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^ExtractorPro” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^EyeNetIE” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^feedfinder” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^FHscan” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^FlashGet” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^flicky” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^g00g1e” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^GetRight” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^GetWeb\!” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^Go\!Zilla” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^Go\-Ahead\-Got\-It” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^grab” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^GrabNet” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^Grafula” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^harvest” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^HMView” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^Image Stripper” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^Image Sucker” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^InterGET” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^Internet Ninja” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^InternetSeer\.com” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^jakarta” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^Java” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^JetCar” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^JOC Web Spider” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^kanagawa” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^kmccrew” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^larbin” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^LeechFTP” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^libwww” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^Mass Downloader” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^microsoft\.url” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^MIDown tool” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^miner” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^Mister PiX” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^MSFrontPage” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^Navroad” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^NearSite” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^Net Vampire” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^NetAnts” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^NetSpider” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^NetZIP” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^nutch” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^Octopus” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^Offline Explorer” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^Offline Navigator” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^PageGrabber” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^Papa Foto” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^pavuk” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^pcBrowser” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^PeoplePal” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^planetwork” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^psbot” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^purebot” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^pycurl” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^RealDownload” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^ReGet” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^Rippers 0” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^sitecheck\.internetseer\.com” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^SiteSnagger” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^skygrid” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^SmartDownload” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^sucker” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^SuperBot” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^SuperHTTP” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^Surfbot” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^tAkeOut” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^Teleport Pro” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^Toata dragostea mea pentru diavola” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^turnit” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^vikspider” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^VoidEYE” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^Web Image Collector” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^WebAuto” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^WebBandit” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^WebCopier” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^WebFetch” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^WebGo IS” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^WebLeacher” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^WebReaper” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^WebSauger” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^Website eXtractor” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^Website Quester” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^WebStripper” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^WebWhacker” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^WebZIP” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^Widow” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^WPScan” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^WWW\-Mechanize” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^WWWOFFLE” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^Xaldon WebSpider” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^Zeus” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “^zmeu” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “360Spider” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “CazoodleBot” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “discobot” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “EasouSpider” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “ecxi” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “GT\:\:WWW” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “heritrix” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “HTTP\:\:Lite” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “HTTrack” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “ia_archiver” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “id\-search” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “IDBot” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “Indy Library” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “IRLbot” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “ISC Systems iRc Search 2\.1” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “LinksCrawler” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “LinksManager\.com_bot” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “linkwalker” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “lwp\-trivial” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “MFC_Tear_Sample” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “Microsoft URL Control” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “Missigua Locator” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “MJ12bot” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “panscient\.com” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “PECL\:\:HTTP” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “PHPCrawl” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “PleaseCrawl” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “SBIder” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “SearchmetricsBot” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “SeznamBot” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “Snoopy” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “Steeler” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “URI\:\:Fetch” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “urllib” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “Web Sucker” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “webalta” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “WebCollage” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “Wells Search II” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “WEP Search” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “XoviBot” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “YisouSpider” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “zermelo” [NC,OR]
    RewriteCond %{HTTP_USER_AGENT} “ZyBorg” [NC,OR]
    # End Abuse Agent Blocking
    # Start Abuse HTTP Referrer Blocking
    RewriteCond %{HTTP_REFERER} “^https?://(?:[^/]+\.)?semalt\.com” [NC,OR]
    RewriteCond %{HTTP_REFERER} “^https?://(?:[^/]+\.)?kambasoft\.com” [NC,OR]
    RewriteCond %{HTTP_REFERER} “^https?://(?:[^/]+\.)?savetubevideo\.com” [NC]
    # End Abuse HTTP Referrer Blocking
    RewriteRule ^.* – [F,L]
    # End HackRepair.com Blacklist, https://pastebin.com/u/hackrepair
    # END iThemes Security – Do not modify or remove this line

Viewing 4 replies - 1 through 4 (of 4 total)

  • It’s not the HackRepair blacklist that is being updated. This isn’t about the HackRepair blacklist at all.

    It’s banned host IPs that are attempted to be written (as Quick Bans) probably as a result of brute force attacks (or possibly too many 404s if the 404 Detection module is enabled).

    However due to a LiteSpeed specific bug in the plugins’ quick ban code there are NO ban rules written, only comment lines.

    So the bug in the quick ban code (quick_ban() class method in the files.php file) needs to be fixed in order to get this working properly on LiteSpeed.

    For now the best thing you can do is configure the plugin properly to prevent any brute force attacks (or fix any 404s that may lead to IP bans or simply disable the 404 Detection module). Have a look at the plugin Logs page to determin the exact origin of the Quick Bans.

    Thread Starter sgarcia513

    (@sgarcia513)

    Thanks for the insight nlpro.

    We don’t have 404 detection enabled. I wasn’t necessarily believing it was HackRepair, specifically, but that the rewrite wasn’t being completed causing the 404 errors. Because of false positives our client asked that User Banning be turned off so I enabled the HackRepair banning to give some sort of protection. My biggest concern was the duplication of those same 4 lines inflating the file.

    I did look at the security logs and, of course, the sites were getting pounded within a minute of each other from various IP addresses but didn’t see anything out of the ordinary.

    I went ahead turned User Banning back on and looked at the htaccess file and it’s much cleaner in regards to those duplicated lines are no longer there.

    My guess is with the constant rewriting of htaccess without adding any IP addresses and the constant duplication of the first four lines there was bound to be a breaking point where the htaccess file wasn’t being fully rewritten properly. Unfortunately, the WP lines are at the very end which instantly breaks a site and requires a manual reset.

    Turning off user banning isn’t typical so it’s a small bug in regards to the htaccess file inflating so quickly. I let the client know that we had to turn User Banning back on and there could be the occasional false positive which we’ll have to deal with on a case-by-case basis.

    Thanks

    Ok, since it is not enabled we can forget about the 404 Detection module. One down …

    I did look at the security logs and, of course, the sites were getting pounded within a minute of each other from various IP addresses but didn’t see anything out of the ordinary.

    With all due respect but when I read the above it sounded to me like you think brute force attacks are normal. It’s not, IMHO it’s unacceptable and should be avoided at all costs ! Repeat after me (like 10 times):

    I don’t want my client’s websites pounded within a minute of each other from various IP addresses !

    I think your clients will agree ??

    So again have a look at the plugin Logs page and figure out which brute force method is being used.

    If it’s xmlrpc.php disable XML-RPC in the WordPress Tweaks module (unless the site really needs it).

    If it’s the login page enable the Hide Backend module.

    If the site is leaking usernames, fix that.

    Do everything you can to prevent client sites from being the low hanging fruit on the internet. This way automated bots move on to other sites to brute force … and you’ll sleep better ??

    Installing the iTSec plugin and just running the Security Check module is not enough. Website security is constant monitoring and taking the right action after any attack …

    Thread Starter sgarcia513

    (@sgarcia513)

    Thanks, this is all great information

    I did look at a random sample size and it looks like they’re all using XML-RPC connection so I’ve disabled them, and plan to for all of our sites since no one uses the mobile version or pingbacks.

    One of the sites already had the Hide Backend plugin being used and was one of the most attacked. Ran through a handful and they were also all using XML-RDC.

    I think this will help tremendously. Also, I’m in the process of setting up LiteSpeed’s Brute Force Protection as well. Hoping to set up multiple obstacles and free up our server resources.

    Thanks again for your help!

Viewing 4 replies - 1 through 4 (of 4 total)
  • The topic ‘Bad htaccess rewrites – causes 404 errors on all subpages’ is closed to new replies.