Of cause we should have noticed, since the amount of spam had dropped to zero. But now that we are receiving thousands of spam comments once again we know that everything is ok ;)

Jesper: One change I really would appreciate in the next Wordpress would be the ability to shift wysiwyg/code editor directly on the edit page

(Now we have to change it on the user prefs page)

Thomas: now = today

Jesper: aggreed! the sooner the better!

I would like always to put my setting to wysiwyg off and turn it on when editing a post individually. Too often I have burnt my fingers and messed up posts containing code, blockquotes or formatted whitespace

If the control were moved to the edit page I could easily write wysiwyg when needed and avoid it when it would mess up my post

Thomas: Yes… it should be possible to tell WordPress that a given post should always open with WYSIWYG off (or on) whenever it is edited… dismissing the default settings

Jesper: exactly!

wysiwyg MUST be off by default because it’s darn destructive! (at least in the current versions of the wysiwyg editors)

Thomas: I wonder if there exists some WP plugins out there that do this already

Jesper: Good question!

Thomas: … or even better doesn’t mess up custom code or HTML

(I mean… hacks WP so it doesn’t)

Jesper: I hate the double/single quotes that WP replaces.

All code examples stop working unless the author manually replaces:

'(single quote) with '

"(double quote) with "

Thomas: I think that there is a simple solution to the quote issue… it must be fairly simple to change that

*searching for a fix*

Sent at 11:04 PM on Friday

Thomas: hmmm it seems to be hard to find

Is the problem that you write a regular quote and WordPress translates it into a html entity or the other way around?

Jesper: The problem is when posting code snippets, HTML or JavaScript for instance. The regular quote (double quote) is automatically translated to the angled double quotes — the same illness as Word does by default.

It’s an illness, in my opinion.

Leave it up to the user to actively change my text

Don’t spoil my code snippets by default.

Making normal quotes in normal text pretty is just not a great tradeoff …

Thomas: depends… the quotes them selves look great when wrapping sentences – but doesn’t work for code – I agree

After this chat I did some googling and found – as I expected – that we are not alone when it comes to the angled quote issue. The quote replacement is actually called “Smart Quotes” in WordPress, and there is a good discussion on the topic over at WordPress Support.

I also found a very good blog-article on the subject by Aaron Russell called “Trials and tribulations of using WordPress to display code syntax“.

What we want…

The most optimal solution would be a WordPress plugin that simply disables smart quotes inside <code>…</code> blocks. To top it of it should also escape the &, > and < characters.

Can you help?

Do you know any plugins that actually do what we would like? Any settings we overlook in WordPress? If not, I guess we will have to convince somebody into changing a future WordPress version.

Technorati Tags: wordpress, editor, wysiwyg, quotes, code, usability, source code

137 new conversations in my mailbox telling me that 137 articles I wrote were hit by spam comments the last 24 hours.

Important lesson: Almost none of the recent posts have spam comments

The screenshot below shows that only 2 of the latest 20 posts have recieved spam comments. So it seems the logic behind it is that spammers need time to harvest links for potential targets.

(click to view entire count on first 40 articles)

No posts newer than 11 days are hit by spam comments. 2 comments on “Spam Comments Dropped 95% Overnight” from December 4th.

So on this particular day, spam comments only hit in after 11 days and should be used to our advantage. There are currently these different strategies to consider:

1. Close old posts for comments
2. Close old posts for comments but keep active posts open
3. Hold back comments on old posts for moderation
4. Hold back comments on old posts for moderation unless posted by a user that has a previously approved comment

Thomas and I talked this over yesterday, and would really prefer number 4, and we can live with number 3. Number 1 and 2 are implemented by a wordpress plugin (i forget the name).

The real difference is in the user experience. 1 and 2 are unacceptable for us, since it will block our users’ valid comments if, say, somebody has an update to an old article or wants to send a trackback from a related post. This really works against the wisdom of crowds principle, that works best if everybody is allowed to comment right away.

Does anybody know a plugin that can do number 3 or 4?.

Wordpress flaws and bugs:

• Three comments were incorrectly held for moderation by Wordpress even though the link limit were set to 99 and the comments clearly did not contain 99 links.
• The interface for marking comments as spam is unproductive. It took me two hours to mark 1111 comments as spam (and de-spamming 6 valid comments). I used “mass edit” screen and checked all. Pressed “mark checked comments as spam”. Then pressed OK to the JavaScript confirmation.
  I don’t know the internal procedures here — but it seems as if wordpress sends the 20 spam marked comments directly to Akismet. And waits for the response before showing me the next page. My workaround here was to first open page 1-20 in 20 different tabs. Then for each of them check all and press “mark checked comments as spam”.
• Recheck Queue for spam is a brilliant feature — but works on the wrong data. The feature should be copied to the “comments” tab. This way, you could force a rerun if your spam filter is out of order for a period of time. Also that would be the usecase of people trying out a spam filter. “wow I got more spam comments than I can handle. Let me try and install a spam filter and see what it can do for me”. In this case, the spam filter not only works forward in time after activation, but also backwards. (obviously there must be a feature to review the past comments marked as spam).
  
• Blank email posts slip through even when the setting does not allow it. I have suspected this to happen also in previous versions of Wordpress. Several of the 1111 comments we recieved during Spam Filter Free Day were with blank email. And that should not be possible because of the setting in Wordpress “Comment author must fill out name and e-mail” (found at Admin>Options>Discussion).
  
  I would expect these comments to be rejected and not even show up at the administrators’ panels. But for some reasons they still appear. Perhaps a tiny bug in Wordpress? Or maybe spammers found a hole? Or are these trackbacks/pingbacks that just look like comments? Or maybe the updated Akismet version 2.1.2 actually deals with this because they added separate tabs for trackbacks and pingbacks?

All in all, we learned a few lessons and found some areas, where wordpress could improve in order to deal with comments that slips through. Total time consumption for removing these comments were 2 hours, which actually surprised us to be very low. But note also that this was not “just” a day without Akismet. In that case, we would probably had to keep comments under surveillance, and remove them during the day. In that case, time consumption would perhaps have been something like 6 hours during a 24 hour period.

As to whether we would do it again as a reoccuring event? Most likely not. We don’t want our regular readers to suffer, so we prefer to keep our blog safe and sound and in good shape, by keeping the guards up. On the other hand, it has been a learning experience to do the Spam Filter Free Day. Not only has our daily spam count decreased (perhaps because spammers have understood that comments that momentarily slip through have no effect on PageRank or similar). Also, we wanted to raise awareness of the bandwith and processing power (machine and human) that spam comments waste every day. And last also do this day as a thank to spamfilters such as Akismet. (heck, comments on other blogs even alledge that the Spam Filter Day is a publicity trick from Akismet… I can strongly deny that as we — thomas and jesper — can take full responsibility for inventing this event. And we are in no way affiliated with Akismet).

I would really appreciate comments with respect to plugins that do not close articles for comments — but in stead holds comments back for moderation. I will subsequently update this article with links.

Technorati Tags: wordpress, akismet, spam, comment spam, spam filter free day.

When we announced the Spam Filter Free Day 18 days ago, we didn’t think that it would draw so much attention. But the amount of comments and buzz around the blogosphere have been really overwhelming. Most notably is that we caught the attention of the comment-spam fighting company of all times: Akismet.

We also recieved our share of warnings from people telling us that this might not be such a good idea after all. I would like to say up front, that we really appreciate this and that we are wiser even now then before the announcement, but we just can’t resist going ahead with the experiment – We might learn even more!

In the next couple of minutes, we’ll remove our barricades, turn the other cheek and just sit, wait, see and learn.

Final remarks:

1. If you post any legit comment on our blog within the next 24 hours we’ll try not to delete it when we do our manual spam-cleanup – but let me just apologize up front if we do.
2. If you subscribe to our comment-feed, you might notice more comments then usual ;)

Related reading:

• Announcement: Spam Filter Free Day
• Letting Spam In For A Day Questions and Answers
• Spam Comments Dropped 95% Overnight

Technorati Tags: spam filter free day, spam, akismet, blogging, comment spam, spam fighting, bandwidth, justaddwater.dk, event, wordpress, no follow, sketch, monty python

Thomas and I just had a conversation about what is going on. Could it be that spammers now give up after reading that comments here really give them no value (or pagerank)? Not only are links removed with Akismet, .htaccess files, blacklists and more. We also instantly remove the very few that slip through.

Or did Akismet modify it’s algorithms? Or did the worlds worst spammers just get blocked centrally somewhere? Or is this just silence before December 15th?

What do you think?

Is anybody else experiencing heavy drops in spam traffic?

Related info:

• Announcement: Spam Filter Free Day
• Letting Spam In For A Day Questions and Answers

Technorati Tags: akismet, spam, comments, justaddwater, blogging, spam-filter-free, spam filter, wordpress

• Akismet Blog: Letting Spam In For A Day
• Blogherald: Will you turn off Akismet?
• Follow along in Technorati blog reactions for Justaddwater.dk

Also to answer some of the comments we got:

Binh Nguyen:
How about a bigger challenge? Keep everything else active and shut down only the spam protection.

That’s a big challenge. But we actually tried that recently when Justaddwater.dk got hacked. The hacker used a security hole in an old version of Wordpress and to upload his malicious code, he disabled all plugins. I discovered that by discovering that Akismet (and all other plugins) was turned off.

Olysses:
You are nuts. I might join you, but let me think about it. I don’t want to spend the whole day deleting spam. Maybe, a quick database backup and restore as if December 15 never happened. That might do the trick.

Thanks for the tip. That might be what we end up doing if the cleanup task will get too big.

Thomas Baekdal:
I don’t understand why you don’t try to mask your comment form, or prevent the spammers from running their scripts.

Purely for accessibility reasons. We discussed that previously And Thomas Watson and I totally agree that it must be possible to fill in a valid comment without using JavaScript in the browser, cellphone, Iphone, whatever. We also discussed that with the previous corporate webmaster at Capgemini (Francois Nonnenmacher), that used a similar technique to get virtually no spam.

Furthermore, we really really want our blog’s spam comments to go back into Akismet, so that Akismet learns and other blogs can benefit. If we reject spam comments at the door, so to speak, Akismet will never learn about it and it won’t get better at rejecting spam. If somebody has a method for rejecting spam (in .htaccess or by looking at 404’s), and at the same time let Akismet know about it, then please let us know.

Thomas Baekdal:
BTW: Just out of curiosity – how much time do you spend on handling spam emails now? That is, the time it takes to look through the spam being filtered + the time spend deleting spam that missed the filter, Whitelisting and blacklisting people etc.?

At the most 30 minutes per day for both Thomas and me. It’s primarily an ad hoc task. I’m looking at what is new, and get an email when a new comment is posted (or slips through the spam filter). Based on that information I either take action to delete it immediately (if it’s spam) or take other actions such as answering real comments, etc.

Michael Clark:
I’m curious: on December 16th, after you reactivate all of your systems, are you going to submit to Akismet the spam that accumulated on the 15th? And are you using a “professional” Akismet account, or a free one?

Free account. We aim at submitting all spam comments to Akismet and apologies to anybody posting a real comment on December 15th. By the way, is there a way to let Akismet traverse comments back in time to see if it can find any spam once we turn it back on? We really could need that to save time :)

JMorris:
Pardon me if I misinterpreted your post, but it read as though you were going to allow automatic approval of comments. If this is the case, do you intend to put into effect some sort of indexing block in place so that search engines don’t pick up the comments and display them in your SERPs?

You understand correctly that we use automatic approval of comments. We have actually discussed pros and cons of this approach. The worst thing about it is that we offend email subscribers once in a while when a spam comment slips through. However, since we manually remove these comments within 10 minutes or so, we consider the most important thing that a discussion can take place instantly without us approving the discussion.

At a later point, if we find it necessary, we might take the step to change the settings, so that a comment from a new mail address is held for moderation, but a known person can comment with automatic approval.

With respect to SERP (search engine result pages), all links by default have the

tag. This means that no spam comments will benefit at all from posting on our blog.

More comments in the original article: Announcement: Spam Filter Free Day

Technorati Tags: akismet, spam, comments, justaddwater, blogging, spam-filter-free, spam filter, wordpress

Today we got our spam comment number 500,000. Pretty scary to think how much energy, computer power, network traffic is wasted on a completely useless activity: To spoil the content with irrelevant comments.

Currently we get somewhere between 2000 and 3000 spam comments each day.

The thing is, that Thomas and I are discussing: We don’t know exactly how much work our spam filter does for us… Granted, 500,000 spam comments is a pretty high number. But we want to know how much pain this saves us for. How much blood and sweat do we.

I have one suggestion: How about declaring spam-filter free day one day a year. What about December 15th, which is available according to wikipedia?

The purpose of Spam Filter Free day is to

• put focus on how much energy, computer power, network traffic, and manual work is wasted by the completely irrelevant comments
• put focus on spam filters and their current effectiveness
• We remove the filter a saturday, where traffic is usually lower (so the annoyance for the end-users will be as little as possible)

For this to work we have to:

• Disable our secret server-validation (that we use before the comment hits wordpress)
• Disable the plugin that emails subscribers when a new comment arrives
• Disable Akismet
• Disable our blacklist that holds comments for moderation if certain words are in the comment
• Disable the function that holds a comment for moderation if it has a certain number of links
• Disable any plugin that makes spammers gain from the fact that we disable our spam filters. (For instance the “no nofollow” or “dofollow” plugin)

Join us if you like, and drop a comment about it here :) And if you do drop a comment here on December 15th, please forgive us if we accidentally mark a correct comment as spam. We will probably have a hard time cleaning up afterwards. If your comment dissappears, contact us as soon as possible.

Related articles:

• 100,000 Blog Spam Comments (Feb 26th)
• Blog Usability: Spam Comments Irritate Subscribers (April 2nd)
• Spam Commenters Are Wasting Brainpower (September 9th)

Technorati Tags: akismet, spam, comments, justaddwater, blogging, spam-filter-free, spam filter, wordpress

Last Thursday we unfortunately had an intruder on Justaddwater.dk. The cracker used a flaw in our blogging software to gain administrative access to it. With this access the cracker placed a secret backdoor on the server.

It seems that the intention of the cracker where to use our server to host illegal copies of movies as part of a larger pirate network. To our luck the technical skills of the cracker where very bad (read: script kiddie) and the attempt to upload files to our server was not successful. The cracker therefore quickly gave up and did not try to access to backdoor afterwards.

We have now upgraded our blogging software to the newest version effectively closing the security hole and have reverted his (or hers?) changes and damage to the system.

We have tried to track his every move in our system, but cannot guarantee that he at some point didn’t get a copy of all our commenter’s e-mail addresses. We are pretty sure he was not after this information, and it does seem like he did not retrieve them, but we are not a 100% sure.

I know some of you use unique e-mail addresses when writing comments on our blog. If you are such a person and ever get any spam on this unique e-mail address, we would very much like to hear from you.

To those of you who subscribe to our newsletter, you can rest assure that your e-mail addresses where never in any danger. They are safely kept at our service-providers and the intruder have not had access to them.

For more information on how we store user information see our Privacy Policy.

Info to fellow WordPress users

The versions of WordPress affected by this security issue are:

• The entire version 2.1.x branch
• In the 2.0.x branch, every version below 2.0.11 is affected

If you are still running WordPress version 2.1.x we urge you to quickly upgrade to a newer version (2.2 or above). If you are running the older 2.0.x branch you can upgrade to version 2.0.11.

All the technical goodies

The bug in question is a serious SQL injection bug affecting

where the attacker can perform any SQL command on your WordPress database (including wiping it out completely!).

In our case the cracker “only” used it to gain administrative WordPress access and updated the WordPress “upload_path” to point to the servers

directory. He then uploaded a PHP script to this folder and added it as a WordPress plugin by manipulating the “active_plugins” option in WordPress. The purpose of this plugin was to lay dormant until a call to a specific URL was detected. This secret URL would then render a backdoor interface instead of the regular justaddwater.dk. You can see the backdoor interface by clicking the thumbnail below:

After looking at the Apache log files we can see what the cracker actually did to gain access and what he used that access to do.

First he made a very strange call:

I have no idea why he called this URL first. My guess first was that this was a way of detecting which version of WordPress we where running, but after investigating I can see that this file exists also in versions not affected by the bug.

Secondly he called the URL affected by the SQL injection bug several times (actually 159!):

Every call looks the same except for the size of the response in bytes following the 200 response code (in the above log entry it’s 2, but sometimes it would be 298 or 299). It would have been during these calls that the cracker gained administrative access to WordPress. The cracker also created a new post that he would later use as a placeholder for the PHP script he where to upload.

Now followed 8 calls to options.php:

Notice how the user-agent changes all the time. This could be different scripts running, set to identify them selfs differently. My guess is that he used these calls to change the “upload_path” option and the “active_plugins” option. He could have done this via SQL injection though.

Then came 10 calls to upload.php, the script used to upload pictures etc. to posts:

My guess is that he used these calls to upload the backdoor script to /tmp. Why he needed 10 I don’t know.

Then came another 15 calls to options.php:

It’s not quite obvious why he made these last calls, except to maybe clean up something he made earlier.

Now came a very strange call to upgrade.php that I can’t figure out why he made:

This script is used to upgrade the WordPress database after installing a new version of WordPress. He might have used some of the previous requests to prepare an “upgrade” script that he could then run this way, but why didn’t he just use the SQL injection technique from before?

Finally he accessed his newly created backdoor, looked inside our

folder, did 3 POSTS (that I suspect where file-upload attempts) and then left never to be seen again.

Technorati Tags: script kiddie, hacking, hacker, cracker, WordPress, security, justaddwater.dk, bug, blogging software, hacked, vulnerability, apache, log files

The number of daily spam comments has grown more than 400% over the last year.

Though Thomas and I are thrilled with how well Akismet works, there are still one serious issue: Every day somewhere between 1 and 10 spam comments slip through. It’s not a big deal for us, as we usually remove them within minutes if they slip though.

Spam comments annoy subscribers

However, the comments that slip through actually annoys the most loyal readers. Our readers can subscribe to new comments on any article, and the spam that slips through actually generates an email to all subscribers.

Furthermore, any spam comments slipping through Akismet will occur in the comment feeds.

I feel really sorry for the subscribers because a subscription then eventually generates an email, and even though we remove the spam comments fast, people still gets irritating, irrelevant spam sent by us to the subscribers. And we really hate that.

I think that this problem is more widespread than it seems, as I myself have recieved some emails from blog articles I’m subscribing to:

As a consequence, Thomas and I will now look for ways to tightening things further than Akismet can do.

Possible solutions

Moderate all comments

We have previously talked about holding back all comments for moderation (or at least comments from unknown email adresses). This is very easy to do — just flip a switch in a Wordpress administration module. One disadvantage with this solution: It adds a time-delay to the blog, and we feel that a blog should be immediate, and a discussion should take place without us acting as gatekeepers for the discussion.

Improve Akismet

We are not in a position to change or improve Akismet directly, but here is a suggestion: Comments should be evaluated better and not just based on if anybody else has marked a comment to spam. It would be cool if Akismet could sometimes “be in doubt” about a comment. What if Akismet could tell Wordpress to hold a comment for moderation if it was likely to be spam. For example some of the comments that slipped though the last month had a Russian or Polish email address or www.yahoo.com as homepage. Some others contained the word casinos, which is a very unlikely word to use on this blog. This is of course not true to any blog, so it would be great marking this as “I’m not sure. please moderate for me”.

This suggestion is probably based on my lack of knowledge on how Akismet really works. But it would of course be ideal to add rules to Akismet in order to improve the program.

Adding a bot-filter in the comment form

We talked about adding one or more form fields and hide them via css, to not disturb screen readers. Then name the field something innocent test that the field is blank when recieved. If not blank, then it is likely that a robot has filled out the field. Not a human, because the field is hidden.

This probably requires some work as we have to try out possibilities for confusing spam bots and see what works and whatnot. There are probably also some disadvantages for people browsing without CSS.

Actually we have already done some work including renaming the wordpress standard form. That worked great at the time, but after 6 months or so, it did not really reduce the spam (we could actually see that in the error log).

Our preliminary conclusion

Eventually we will probably turn on comment moderation, as it is the only 100% guarantee that spam comments don’t slip through to our subscribers and loyal readers. Before we do that (and to avoid the time-delay drawbacks), we will experiment on making a better bot-filter in the comment form.

More info

• Justaddwater.dk: 100,000 Blog Spam Comments, February 6, 2007
• Justaddwater.dk: Blog Usability: Avoid Spam Comments (Part 2), June 2006
• Justaddwater.dk: Blog Usability: Avoid Spam Comments, June 2006

PS. We have previously claimed that we won’t use CAPTCHAs or encode comment form with JavaScript, and for now, we will stand by that decision, in order to avoid usability and accessibility issues.

Technorati Tags: akismet, spam, blog usability, usability, blogging, wordpress

Sorry to have you waiting. There are two main reasons why I have kept it private:

1. I wanted to check that the description worked well for me. And today I got my new laptop.
2. It surprised me that a password protected article shows up on the front-page, RSS feeds, etc.

As I found out the original file was published, I felt that I could not remove it even though I was not ready to publish entirely.

WordPress flaw or feature?

I think its hard to imagine that Wordpress publish password protected articles on purpose (even though it’s only the headline).

If this really is a feature, I would prefer another feature: To be able to publish a “stealth” article, password protected that should not show up on the frontpage or in RSS feeds. Is this feature already available? If so, there is some bad usability involved preventing me from using the password protection as I wanted to :)

Now, the article  “Prototyping Ruby on Rails Checklist to Get Started”

Technorati Tags: ruby on rails, prototyping, rapid prototyping, preparation, rails, wordpress, usability, stealth, password, protection, rss, frontpage, private, privacy, blogging