Sorry, there seems to be a problem.

Or I’m sent to infinite redirect loop. None of their self-help pages are useful or have this error mentioned.

Apparently if you get such message, you should read this message in opposite way:

“Sorry (or you should be sorry), there seems to be a problem. This service is looking for you and you seem to be temporarily unavailable. We’re not working on this at all, it may show up on some report at the end of the year though as lost revenue, sad, we see you spent few hundred dollars on our services, so you trust us, but of course we have enough money from everyone else. You may try again in few minutes, few hours or few days, it won’t help you though. Though we appreciate your patience, you should start researching and debugging this yourself instead, thank you.”

I cleared all Google cookies on my browser and it started working. Apparently this has happened to others too.

Update: again…

Oh well, I know I shouldn’t poke directly at people, but they deserve that sometimes (at least in my very personal opinion). Heck, I even gave 12h window for this not to be hot-headed opinion.

Those who followed MySQL at facebook development probably know how much we focus on actual performance on top of mixed-composition I/O devices (flashcache, etc) – not just retreating to comfortable zone of in-memory (or in-pure-flash) data.

I feel somewhat sad that I have to put this truism out here – disks are way more cost efficient, and if used properly can be used to facilitate way more long-term products, not just real time data. Think Wikipedia without history, think comments that disappear on old posts, together with old posts, think all 404s you hit on various articles you remember from the past and want to read.

Building the web that lasts is completely different task from what academia people imagine building the web is.

I already had this issue with other RDBMS pioneer (there’s something in common among top database luminaries) – he also suggested that disks are things of the past and now everything has to be in memory, because memory is cheap. And data can be whatever unordered clutter, because CPUs can sort it, because CPUs are cheap.

They probably missed Al Gore message. Throwing more and more hardware without fine tuning for actual operational efficiency requirements is wasteful and harms our planet. Yes, we do lots of in-memory efficiency work, so that we reduce our I/O, but at the same time we balance the workload so that I/O subsystem provides as efficient as possible delivery of the long tail.

What happens in real world if one gets 2x efficiency gain? Twice more data can be stored, twice more data intensive products can be launched.
What happens in academia of in-memory databases, if one gets 2x efficiency gain? A paper.
What happens when real world doesn’t read your papers anymore? You troll everyone via GigaOM.

Though sure, there’s some operational overhead in handling sharding and availability of MySQL deployments, at large scale it becomes somewhat constant cost, whereas operational efficiency gains are linear.

Update: Quite a few people pointed out that I was dissing a person who has done incredible amount of contributions, or that I’m anti-academia. I’m not, and I extremely value any work that people do wherever they are, albeit I do apply critical thinking to whatever they speak.

In my text above (I don’t want to edit and hide what I said) I don’t mean that “a paper” is useless. Me and my colleagues do read papers and try to understand the direction of computer science and how it applies to our work (there are indeed various problems yet to solve). I’d love to come up with something worth a paper (and quite a few of my colleagues did).

Still, if someone does not find that direction useful, there’s no way to portray them the way the original GigaOM article did.

I feel ashamed that I ever wanted you to support 4.0->5.1 replication, and apologize for that. I really understand that it was really egoistic of me even to consider you should be involved in this.

I even understand that 5.0 is running out of active support (I’m not questioning that you’ll stop supporting 4.1 entirely too), and you’ll stop doing pretty much anything to 5.0, except “critical security fixes” (w00t, I managed to get one into 4.1, 8 year old MITM flaw :).

I really understand that supporting more than one release is very very very difficult, and people should do only adjacent version upgrade.

I’m not asking you much, but, maybe you could then support 5.1 to 5.1 replication? I don’t want much, just:

• Gracefully recover after slave crashes.
• Don’t have single serial reading of pages for replication stream as a bottleneck – either read-ahead properly (you can do that with RBR!!!), or apply events in parallel (you can do that with RBR too!)
• Allow to edit replication filters without restarting servers.
• Allow to enable and disable binary logging of events received from master, as well as enabling and disabling binary logging without restarting the instance.

I hope it isn’t too much too ask! It is just supported replication between two same version instances.

Thanks!

I will do this to rest of your code, if you continue breaking replication for me.

– Domas

This particular benchmark has two favorite parts, that go with each other together really well:

I didnt change absolutely any parameters for the servers, eg didn’t change the innodb_buffer_pool_size or key_buffer_size.

And..

If you need speed just to fetch a data for a given combination or key, Redis is a solution that you need to look at. MySQL can no way compare to Redis and Memcache. …

Seriously, how does one repay for all the damage of such idiotic benchmarks?

P.S. I’ve ranted at benchmarks before, and will continue doing so.

I don’t know how effective your scare-mongering cash-extortion tactics are, but they don’t really help neither your users, nor vendors, nor anyone else.

It all starts when major vulnerability databases start authoritatively spouting out crap like this:

A vulnerability has been reported in MySQL, which can be exploited to compromise a vulnerable system.
The vulnerability is caused due to an unspecified error and can be exploited to cause a buffer overflow. (Secunia)

Or crap like this:

MySQL is prone to a buffer-overflow vulnerability because if fails to perform adequate boundary checks on user-supplied data.
An attacker can leverage this issue to execute arbitrary code within the context of the vulnerable application. Failed exploit attempts will result in a denial-of-service condition. (Securityfocus)

Of course, there must be some reason to publish such claims, right? Of course, if you brought in cash for all these security vendors, they’d definitely tell you that in future they will give you all available updates, but for now:

Due to the very limited available information, it is not possible to suggest an effective workaround.

Why would one post such crap? Because people want to sell things, like vulnerability databases full of rubbish:

A working commercial exploit is available through Intevydis. This exploit is not otherwise publicly available or known to be circulating in the wild (Securityfocus)

This is how the Security Vendor that unleashed this all classifies the vulnerability:

• Name: MySQL 5.x exploit
• Status: 0day
• Details: Remote buffer overflow exploit. Tested on Debian Linux 5.0 with mysql-server 5.0.51a-24+lenny1
• Listener: LINUXMOSDEF
• Platform: Linux x86

If you’d look it up, Debian just had security advisory DSA-1877-1 (released at September 2nd, one day before Secunia and Securityfocus went live with their stuff), which tells us about denial of service and execution of arbitrary code possibility. The +lenny2 package fixes it, the +lenny1 package is vulnerable. At this point, I don’t really have anything against Debian security people – Linux distribution security contacts are always eager to communicate, share, discuss, and, well, admit failures – and as they are not in money extortion business, it is very easy to forgive them. Still, the advisory mentions “potential execution of arbitrary code via format string specifiers”. See, original exploit at milw0rm did mention “format string vulnerability”, which is source of  [easily understandable] confusion here.

Various C libraries allow passing %n format specifier to printf() calls, which is:

The number of characters written so far is stored into the integer indicated by the int * (or variant) pointer argument.  No argument is converted.

This means that if you allow someone to pass a format string, he can overwrite memory of your application with specially crafted data (though, it isn’t that trivial to exploit it). MySQL though, as it has to be very portable, has its own version of printf, to avoid any OS-specific behaviors, and that implementation does not have %n, which means by passing arbitrary format string you cannot execute arbitrary code. Phew.

So, what is the bug? Let me present you Bug#45790:

• If server has General Query Log enabled (thats very very low percentage of systems out there) and….
• User has right to create databases (which isn’t a right given away to every user out there….) then…
• He can shut down (well, crash) the server!

Summary of this whole security catastrophe in practice would be:

System administrator can shut down MySQL server

My heart pounds and I hurry to upgrade every machine to 5.0.84 (sarcasm aside, do it anyway, it has great fixes ;-).

P.S. I may be wrong, and there is an exploit which will convert your database clusters into botnet zombies, capable of much more than regular botnets – imagine all the multicore servers attached to SANs completely dominating the world. Scary. Thats why we try to work on all security threats – even if it takes quite a few hours too long for what it deserves.

When carrying out more database benchmarking, but this time with PostgreSQL, XFS and Btrfs were too slow to even complete this test, even when it had been running for more than an hour for a single run. Between EXT3, EXT4, and NILFS2, the fastest file-system was EXT3 and then its successor, EXT4, was slightly behind that. Far behind the position of EXT4 were NILFS2 and then Btrfs and XFS.

There were few other benchmarks, e.g. SQLite showed ‘bad performance’ on XFS and Btrfs.

*clear throat*

Dear benchmarkers, don’t compare apples and oranges. If you see differences between benchmarks, do some very very tiny research, and use some intellect, that you, as primates, do have. If database tests are slowest on filesystems created by Oracle (who know some stuff about systems in general) or SGI (who, despite giving away their campus to Google, still have lots of expertise in the field), that can indicate, that your tests are probably flawed somewhere, at least for that test domain.

Now, probably you’ve heard about such thing as ‘data consistency’. That is something what database stack tries to ensure, sometimes at higher costs, like not trusting volatile caches, enforcing certain write orders, depending on acknowledgements by underlying hardware.

So, in this case it wasn’t “benchmarking file systems”, it was simply, benchmarking “consistency” against “no consistency”. But don’t worry, most benchmarks have such flaws – getting numbers but not understanding them makes results much more interesting, right?

Oh, and… thanks for few more misguided people.

Now, Ubuntu provides you with AppArmor (my new favorite security thing :) profile that doesn’t suck either, and Debian is a bit rude:
There is ‘tasksel’ tool, which allows you to install groups of packages. Debian has a task ‘Database server’ which will install PostgreSQL. Ubuntu is more politically correct and has renamed that task to “PostgreSQL server”.

MySQL has to do quite some work to repair its image in OS-engineer-geek mindsets, these people tend to love PG more, and once project is community-led, most active participants win the decisive power.

Quite important to look at versions – Ubuntu Intrepid carries 5.0.67, and Debian carries 5.0.51. As I’m way more exposed to everyday issues within MySQL, I value distributions that don’t just backport security fixes, but use MySQL bugfix releases way more proactively. The difference between those two versions has a nearly full year of performance, stability and feature maturity work (and thats really LOTS of real hard intensive work by my teammates). Debian will probably stay at that exact version, unless one uses ‘testing’ or ‘unstable’ packages – so over time the gap will just widen.

Oh, and why I ended up writing this today – Ubuntu/Canonical people were really nice and open, and tried to get as much as possible of feedback to make MySQL shine on their distribution. Gotta say, they’re already leading :)

• It’s a nightmare. Only Mozilla Thunderbird gets more disrespect from me. – Lead developer Jason describes the software they use, mediawiki.
• Can we like anonymize some stuff and submit Mediawiki to WorseThanFailure? – Technical liasion [sic] Zachary suggests.

Of course, being forced to run open-source package from greatest ‘rival’ is pain oh pain. Citizendium team even forked software, called it CaesarWiki.
This is how improvements to the fork are described:

Well, hypothetically, we can do whatever we want in terms of improving MediaWiki, including working on the difference engine.
However, I think that it’s more likely that any changes in that area will filter down from work done by the MediaWiki team.
They have a lot more developer time (in developer-hours/month) and a lot more expertise with MediaWiki.

Of course, having paid lead developer not understand core principles of how software functions (disrespect, remember?) doesn’t help with real improvements. Of course, half a year ago, big work was ahead:

Ideally, I would like to rewrite mediawiki from the ground up in OO style. Since that may not work well, the best way is to wrap it in a bow and let the “present” develop into something pretty over time.

Xoops was given as an example of package that scales, has some security, even uses caching, so integrating with MediaWiki would make it scale. Thats sure way forward. Of course, one of biggest mistakes Wikipedia folks has made is LAMP choice:

To not box ourself in like Wikipedia has done with Mediawiki, PHP and MySQL, we need to pursue modular, easy to use and easy to maintain and update solutions. No one needs network and system admins spinning dinner plates on sticks all day.

It is quite difficult to understand how people who never talked to us know about our operations that much. Back when this was written, Wikipedia had one full-time employee working on the system, few others did the work whenever they (we) wished, and that usually was creative (of course, sometimes artistic) work. Anyway, to run away from evil MySQL to PG, this set of arguments was used:

Disadvantage: no years of heavy use to test it. Advantage: fewer workarounds, easier to scale overall, incredibly knowledgeable community ready to help out.

Of course, at Wikipedia we failed to scale. Now what made me slightly envious, is discussion about security and operating personnel – having a pool of developers scattered around the world, with floating 24/7 schedule is priceless, we really can’t afford that at Wikipedia – at one moment all of us were in Europe, now just Brion is sitting in Florida (what is not that far away either).

Anyway, though I believe in Wikipedia evolution more than in Citizendium revolution, I wouldn’t reject advises – the project may be quite interesting, and if content can be reused on other projects, it just adds value to the Web. Probably we’re rookies in software engineering, but there has been long path to build Wikipedia platform. Some of us learnt technologies used specifically for the project. I’m not sure we did earn the disrespect we’re getting, but I still think that antagonism is harming Citizendium, not us.