Of cause we should have noticed, since the amount of spam had dropped to zero. But now that we are receiving thousands of spam comments once again we know that everything is ok ;)

137 new conversations in my mailbox telling me that 137 articles I wrote were hit by spam comments the last 24 hours.

Important lesson: Almost none of the recent posts have spam comments

The screenshot below shows that only 2 of the latest 20 posts have recieved spam comments. So it seems the logic behind it is that spammers need time to harvest links for potential targets.

(click to view entire count on first 40 articles)

No posts newer than 11 days are hit by spam comments. 2 comments on “Spam Comments Dropped 95% Overnight” from December 4th.

So on this particular day, spam comments only hit in after 11 days and should be used to our advantage. There are currently these different strategies to consider:

1. Close old posts for comments
2. Close old posts for comments but keep active posts open
3. Hold back comments on old posts for moderation
4. Hold back comments on old posts for moderation unless posted by a user that has a previously approved comment

Thomas and I talked this over yesterday, and would really prefer number 4, and we can live with number 3. Number 1 and 2 are implemented by a wordpress plugin (i forget the name).

The real difference is in the user experience. 1 and 2 are unacceptable for us, since it will block our users’ valid comments if, say, somebody has an update to an old article or wants to send a trackback from a related post. This really works against the wisdom of crowds principle, that works best if everybody is allowed to comment right away.

Does anybody know a plugin that can do number 3 or 4?.

WordPress flaws and bugs:

• Three comments were incorrectly held for moderation by WordPress even though the link limit were set to 99 and the comments clearly did not contain 99 links.
• The interface for marking comments as spam is unproductive. It took me two hours to mark 1111 comments as spam (and de-spamming 6 valid comments). I used “mass edit” screen and checked all. Pressed “mark checked comments as spam”. Then pressed OK to the JavaScript confirmation.
  I don’t know the internal procedures here — but it seems as if wordpress sends the 20 spam marked comments directly to Akismet. And waits for the response before showing me the next page. My workaround here was to first open page 1-20 in 20 different tabs. Then for each of them check all and press “mark checked comments as spam”.
• Recheck Queue for spam is a brilliant feature — but works on the wrong data. The feature should be copied to the “comments” tab. This way, you could force a rerun if your spam filter is out of order for a period of time. Also that would be the usecase of people trying out a spam filter. “wow I got more spam comments than I can handle. Let me try and install a spam filter and see what it can do for me”. In this case, the spam filter not only works forward in time after activation, but also backwards. (obviously there must be a feature to review the past comments marked as spam).
  
• Blank email posts slip through even when the setting does not allow it. I have suspected this to happen also in previous versions of WordPress. Several of the 1111 comments we recieved during Spam Filter Free Day were with blank email. And that should not be possible because of the setting in WordPress “Comment author must fill out name and e-mail” (found at Admin>Options>Discussion).
  
  I would expect these comments to be rejected and not even show up at the administrators’ panels. But for some reasons they still appear. Perhaps a tiny bug in WordPress? Or maybe spammers found a hole? Or are these trackbacks/pingbacks that just look like comments? Or maybe the updated Akismet version 2.1.2 actually deals with this because they added separate tabs for trackbacks and pingbacks?

All in all, we learned a few lessons and found some areas, where wordpress could improve in order to deal with comments that slips through. Total time consumption for removing these comments were 2 hours, which actually surprised us to be very low. But note also that this was not “just” a day without Akismet. In that case, we would probably had to keep comments under surveillance, and remove them during the day. In that case, time consumption would perhaps have been something like 6 hours during a 24 hour period.

As to whether we would do it again as a reoccuring event? Most likely not. We don’t want our regular readers to suffer, so we prefer to keep our blog safe and sound and in good shape, by keeping the guards up. On the other hand, it has been a learning experience to do the Spam Filter Free Day. Not only has our daily spam count decreased (perhaps because spammers have understood that comments that momentarily slip through have no effect on PageRank or similar). Also, we wanted to raise awareness of the bandwith and processing power (machine and human) that spam comments waste every day. And last also do this day as a thank to spamfilters such as Akismet. (heck, comments on other blogs even alledge that the Spam Filter Day is a publicity trick from Akismet… I can strongly deny that as we — thomas and jesper — can take full responsibility for inventing this event. And we are in no way affiliated with Akismet).

I would really appreciate comments with respect to plugins that do not close articles for comments — but in stead holds comments back for moderation. I will subsequently update this article with links.

Technorati Tags: wordpress, akismet, spam, comment spam, spam filter free day.

When we announced the Spam Filter Free Day 18 days ago, we didn’t think that it would draw so much attention. But the amount of comments and buzz around the blogosphere have been really overwhelming. Most notably is that we caught the attention of the comment-spam fighting company of all times: Akismet.

We also recieved our share of warnings from people telling us that this might not be such a good idea after all. I would like to say up front, that we really appreciate this and that we are wiser even now then before the announcement, but we just can’t resist going ahead with the experiment – We might learn even more!

In the next couple of minutes, we’ll remove our barricades, turn the other cheek and just sit, wait, see and learn.

Final remarks:

1. If you post any legit comment on our blog within the next 24 hours we’ll try not to delete it when we do our manual spam-cleanup – but let me just apologize up front if we do.
2. If you subscribe to our comment-feed, you might notice more comments then usual ;)

Related reading:

• Announcement: Spam Filter Free Day
• Letting Spam In For A Day Questions and Answers
• Spam Comments Dropped 95% Overnight

Technorati Tags: spam filter free day, spam, akismet, blogging, comment spam, spam fighting, bandwidth, justaddwater.dk, event, wordpress, no follow, sketch, monty python

Thomas and I just had a conversation about what is going on. Could it be that spammers now give up after reading that comments here really give them no value (or pagerank)? Not only are links removed with Akismet, .htaccess files, blacklists and more. We also instantly remove the very few that slip through.

Or did Akismet modify it’s algorithms? Or did the worlds worst spammers just get blocked centrally somewhere? Or is this just silence before December 15th?

What do you think?

Is anybody else experiencing heavy drops in spam traffic?

Related info:

• Announcement: Spam Filter Free Day
• Letting Spam In For A Day Questions and Answers

Technorati Tags: akismet, spam, comments, justaddwater, blogging, spam-filter-free, spam filter, wordpress

• Akismet Blog: Letting Spam In For A Day
• Blogherald: Will you turn off Akismet?
• Follow along in Technorati blog reactions for Justaddwater.dk

Also to answer some of the comments we got:

Binh Nguyen:
How about a bigger challenge? Keep everything else active and shut down only the spam protection.

That’s a big challenge. But we actually tried that recently when Justaddwater.dk got hacked. The hacker used a security hole in an old version of WordPress and to upload his malicious code, he disabled all plugins. I discovered that by discovering that Akismet (and all other plugins) was turned off.

Olysses:
You are nuts. I might join you, but let me think about it. I don’t want to spend the whole day deleting spam. Maybe, a quick database backup and restore as if December 15 never happened. That might do the trick.

Thanks for the tip. That might be what we end up doing if the cleanup task will get too big.

Thomas Baekdal:
I don’t understand why you don’t try to mask your comment form, or prevent the spammers from running their scripts.

Purely for accessibility reasons. We discussed that previously And Thomas Watson and I totally agree that it must be possible to fill in a valid comment without using JavaScript in the browser, cellphone, Iphone, whatever. We also discussed that with the previous corporate webmaster at Capgemini (Francois Nonnenmacher), that used a similar technique to get virtually no spam.

Furthermore, we really really want our blog’s spam comments to go back into Akismet, so that Akismet learns and other blogs can benefit. If we reject spam comments at the door, so to speak, Akismet will never learn about it and it won’t get better at rejecting spam. If somebody has a method for rejecting spam (in .htaccess or by looking at 404′s), and at the same time let Akismet know about it, then please let us know.

Thomas Baekdal:
BTW: Just out of curiosity – how much time do you spend on handling spam emails now? That is, the time it takes to look through the spam being filtered + the time spend deleting spam that missed the filter, Whitelisting and blacklisting people etc.?

At the most 30 minutes per day for both Thomas and me. It’s primarily an ad hoc task. I’m looking at what is new, and get an email when a new comment is posted (or slips through the spam filter). Based on that information I either take action to delete it immediately (if it’s spam) or take other actions such as answering real comments, etc.

Michael Clark:
I’m curious: on December 16th, after you reactivate all of your systems, are you going to submit to Akismet the spam that accumulated on the 15th? And are you using a “professional” Akismet account, or a free one?

Free account. We aim at submitting all spam comments to Akismet and apologies to anybody posting a real comment on December 15th. By the way, is there a way to let Akismet traverse comments back in time to see if it can find any spam once we turn it back on? We really could need that to save time :)

JMorris:
Pardon me if I misinterpreted your post, but it read as though you were going to allow automatic approval of comments. If this is the case, do you intend to put into effect some sort of indexing block in place so that search engines don’t pick up the comments and display them in your SERPs?

You understand correctly that we use automatic approval of comments. We have actually discussed pros and cons of this approach. The worst thing about it is that we offend email subscribers once in a while when a spam comment slips through. However, since we manually remove these comments within 10 minutes or so, we consider the most important thing that a discussion can take place instantly without us approving the discussion.

At a later point, if we find it necessary, we might take the step to change the settings, so that a comment from a new mail address is held for moderation, but a known person can comment with automatic approval.

With respect to SERP (search engine result pages), all links by default have the

tag. This means that no spam comments will benefit at all from posting on our blog.

More comments in the original article: Announcement: Spam Filter Free Day

Technorati Tags: akismet, spam, comments, justaddwater, blogging, spam-filter-free, spam filter, wordpress

Today we got our spam comment number 500,000. Pretty scary to think how much energy, computer power, network traffic is wasted on a completely useless activity: To spoil the content with irrelevant comments.

Currently we get somewhere between 2000 and 3000 spam comments each day.

The thing is, that Thomas and I are discussing: We don’t know exactly how much work our spam filter does for us… Granted, 500,000 spam comments is a pretty high number. But we want to know how much pain this saves us for. How much blood and sweat do we.

I have one suggestion: How about declaring spam-filter free day one day a year. What about December 15th, which is available according to wikipedia?

The purpose of Spam Filter Free day is to

• put focus on how much energy, computer power, network traffic, and manual work is wasted by the completely irrelevant comments
• put focus on spam filters and their current effectiveness
• We remove the filter a saturday, where traffic is usually lower (so the annoyance for the end-users will be as little as possible)

For this to work we have to:

• Disable our secret server-validation (that we use before the comment hits wordpress)
• Disable the plugin that emails subscribers when a new comment arrives
• Disable Akismet
• Disable our blacklist that holds comments for moderation if certain words are in the comment
• Disable the function that holds a comment for moderation if it has a certain number of links
• Disable any plugin that makes spammers gain from the fact that we disable our spam filters. (For instance the “no nofollow” or “dofollow” plugin)

Join us if you like, and drop a comment about it here :) And if you do drop a comment here on December 15th, please forgive us if we accidentally mark a correct comment as spam. We will probably have a hard time cleaning up afterwards. If your comment dissappears, contact us as soon as possible.

Related articles:

• 100,000 Blog Spam Comments (Feb 26th)
• Blog Usability: Spam Comments Irritate Subscribers (April 2nd)
• Spam Commenters Are Wasting Brainpower (September 9th)

Technorati Tags: akismet, spam, comments, justaddwater, blogging, spam-filter-free, spam filter, wordpress

This one was caught in the moderation queue (and would probably be caught with Akismet otherwise). For entertainment purposes, here is the text I recieved:

Anikrichard | anlikivanna.80@mail.ru | wwwwww.com | IP: 72.9.235.218

hello , my name is Richard and I know you get a lot of spammy comments ,
I can help you with this problem . I know a lot of spammers and I will ask them not to post on your site. It will reduce the volume of spam by 30-50% .In return Id like to ask you to put a link to my site on the index page of your site. The link will be small and your visitors will hardly notice it , its just done for higher rankings in search engines. Contact me icq 454528835 or write me tedirectory(at)yahoo.com , i will give you my site url and you will give me yours if you are interested. thank you

However, I think the best one Idiot spammers found by Donncha O Caoimh:

This comment was posted automatically using Blog Comment Poster. Check out its site to learn more and start building backlinks to your websites today.

Remember the Seti project, where you could make your computer processor available for other projects when you’re not using the computer. We could probably eliminate global warming and poverty in a few weeks. Find a cure to cancer, Ebola and AIDS before christmas if we could just tab into spam commenters brains. What a waste of brainpower:)

Related posts:

• Blog Usability: Spam Comments Irritate Subscribers
• Spam Blog Posted 182 Articles in One Day
• 100,000 Blog Spam Comments
• Blog Usability: Avoid Spam Comments (part 1) (Part 2)

Technorati Tags: spam comments, spam, comment spam, brainpower, idiot

The number of daily spam comments has grown more than 400% over the last year.

Though Thomas and I are thrilled with how well Akismet works, there are still one serious issue: Every day somewhere between 1 and 10 spam comments slip through. It’s not a big deal for us, as we usually remove them within minutes if they slip though.

Spam comments annoy subscribers

However, the comments that slip through actually annoys the most loyal readers. Our readers can subscribe to new comments on any article, and the spam that slips through actually generates an email to all subscribers.

Furthermore, any spam comments slipping through Akismet will occur in the comment feeds.

I feel really sorry for the subscribers because a subscription then eventually generates an email, and even though we remove the spam comments fast, people still gets irritating, irrelevant spam sent by us to the subscribers. And we really hate that.

I think that this problem is more widespread than it seems, as I myself have recieved some emails from blog articles I’m subscribing to:

As a consequence, Thomas and I will now look for ways to tightening things further than Akismet can do.

Possible solutions

Moderate all comments

We have previously talked about holding back all comments for moderation (or at least comments from unknown email adresses). This is very easy to do — just flip a switch in a WordPress administration module. One disadvantage with this solution: It adds a time-delay to the blog, and we feel that a blog should be immediate, and a discussion should take place without us acting as gatekeepers for the discussion.

Improve Akismet

We are not in a position to change or improve Akismet directly, but here is a suggestion: Comments should be evaluated better and not just based on if anybody else has marked a comment to spam. It would be cool if Akismet could sometimes “be in doubt” about a comment. What if Akismet could tell WordPress to hold a comment for moderation if it was likely to be spam. For example some of the comments that slipped though the last month had a Russian or Polish email address or www.yahoo.com as homepage. Some others contained the word casinos, which is a very unlikely word to use on this blog. This is of course not true to any blog, so it would be great marking this as “I’m not sure. please moderate for me”.

This suggestion is probably based on my lack of knowledge on how Akismet really works. But it would of course be ideal to add rules to Akismet in order to improve the program.

Adding a bot-filter in the comment form

We talked about adding one or more form fields and hide them via css, to not disturb screen readers. Then name the field something innocent test that the field is blank when recieved. If not blank, then it is likely that a robot has filled out the field. Not a human, because the field is hidden.

This probably requires some work as we have to try out possibilities for confusing spam bots and see what works and whatnot. There are probably also some disadvantages for people browsing without CSS.

Actually we have already done some work including renaming the wordpress standard form. That worked great at the time, but after 6 months or so, it did not really reduce the spam (we could actually see that in the error log).

Our preliminary conclusion

Eventually we will probably turn on comment moderation, as it is the only 100% guarantee that spam comments don’t slip through to our subscribers and loyal readers. Before we do that (and to avoid the time-delay drawbacks), we will experiment on making a better bot-filter in the comment form.

More info

• Justaddwater.dk: 100,000 Blog Spam Comments, February 6, 2007
• Justaddwater.dk: Blog Usability: Avoid Spam Comments (Part 2), June 2006
• Justaddwater.dk: Blog Usability: Avoid Spam Comments, June 2006

PS. We have previously claimed that we won’t use CAPTCHAs or encode comment form with JavaScript, and for now, we will stand by that decision, in order to avoid usability and accessibility issues.

Technorati Tags: akismet, spam, blog usability, usability, blogging, wordpress

At first sight, the trackback looked fairly real. Se screenshot below.

Just one thing caught my eye: Who’s Jeremy?

I went to the site — it looked reasonable at a glance. A look at the categories (oh, it’s called “associates” but is placed where normal blogs have categories): “shop entertainment”, “shop outdoor sports”, etc for 18 categories — all categories point away from the blog to external sites, for instance www.shop4outdoorsports.com.

A quick look at the blog postings showed all had the exact same wording:

<blockquote>[random name] wrote an interesting post today on [this place is blank]
Here’s a quick excerpt:</blockquote>

(my notes in [square brackets] )

Here is the screenshot from the April archives page (the blog is placed on a server that’s still on April 1st (I’m in Denmark at central European time). The interesting fact here is that there are currently 182 postings today.

Quite impressive — if it wasn’t for the fact that all posts are identical (with a different random name, quote and link.

Previously we have written about our spam comment countermeasures in “how to avoid spam comments” (part 1, part 2), and the last 2 months since we passed 100,000 spam comments, we have had another 57,000 spam comments.

For now, I’ve marked the comment as a spam comment, hoping that Akismet will learn from this. What are your experience with this kind of spam? Any other countermeasures that you can recommend?

PS. As I’ve been writing this, the spam blog added another ten articles.

Technorati Tags: akismet, spam, comments, spam blog, spam comment blog

We have for example been blocking certain keywords that we found where common in spam that slipped through Akismet, or we have been changing the standard WordPress URL for posting comments. Finally we have incorporated a little htaccess hack that validates the posters referral URL. This should of cause be our own domain justaddwater.dk, but many spammers actually put some garbage into this header field – and we can then easily block it directly in the Apache web server before it even reaches WordPress. UPDATE: We also allow empty referrals (thanks Håvard, for pointing this out)

8 months ago we had over a period of also 8 months received 6,517 comments totally. Of them only 348 where legit comments! Since then we have received approximately 100,800 comments. 389 where new legit comments and 7,317 was caught in the htaccess filter we applied in July 2006. This translates to 99.6% spam comments.

Technorati Tags: WordPress, Apache, httpd, htaccess, filter, spam, blogging, Akismet, usability, comments, trackbacks, justaddwater.dk