Skip to main content

Tizra gets faster

Non-technical summary: things are lots faster at Tizra sites and admin tools. There's certainly more to do, but we've got more tricks up our sleeves! Because the big current speed boost is related to one cause, and it took me a while to track down, the geek appendage to this post describes what we found and how we fixed it.

Geekly details

I spent a bunch of time last week looking at system performance. As we've been adding customers and usage, we were beginning to feel the pinch. Performance always varies, but the range of response times was getting wider as things slowed, leading me to think that there might be some systemic issues that would give us a quick improvement (and indeed there was some Linux tuning that helped a bit). But data access seemed to be the real issue, so I spent a bunch of time looking into hibernate, and our caching and querying, and then wound up spending a day or so basically watching all the queries go through Postgres. And you know what? most of them seemed much slower than they should be, even though they are pretty hairy.

Of course, the next step was to check for database indexes, and how the query plans were using them. But in hand testing the plans looked good, and the indexes were sensible. But when run by hand the queries were also significantly faster than when hibernate ran them! This was much easier to see now that we have a live load, which is inevitably different from a test setup. So why the difference? Postgres was ignoring our indexes only when Tizra publisher made the queries.

Turns out that there's an old bug in Postgres where it would ignore indexes on bigint fields in prepared statements unless there was an explicit data type cast. (That type confusion was an obscure result of skew between Postgresql and the SQL standard.) And that was the behavior I was seeing, even though we were using a much more recent vintage of all the software. This was terrible for us, because we have a multi-tenant publishing system for large document collections and we use bigints as primary object identifiers!

So, why the old problem if the bug is gone, and we are not using postgres 7? It turns out that we dynamically build those hairy queries, in HQL (hibernate query language), using the String trick. But nowadays instead of making your indexes work, it breaks them! The differences are invisible in the SQL. It turned out that we were in a version "donut hole." Our database was recent enough so the String trick worked the opposite way (preventing fast queries for our prepared statements), but the JDBC driver wasn't making the calls in the right way to make the old trick work. End result: we're now running the latest JDBC driver with compatibility options set while we update our hairy query generator. And now we can really start tuning our setup!

If the web had not provided the history of the old bug, I would have had a much worse time even knowing where to look to find our somewhat subtle configuration issue. So enjoy the speedup, I sure am!

Comments

Anonymous said…
This is pretty wierd, any chance you can post the exact version numbers of Postgres, Hibernate, and JDBC involved?

Popular posts from this blog

Using XML to Create a Better Online Reading Experience for the American Payroll Association

Congrats to the American Payroll Association on their recent launch of XML-based publications on Tizra!  Thanks to this collaboration, APA's authoritative books for payroll professionals are now available in crisp, reflowable HTML, creating a user experience that feels like a truly digital native product, rather than a conversion from print. XML-based publishing also creates a better mobile reading experience, supports more precise search and navigation, and opens the door to better accessibility for users with low vision and other disabilities. Our partners at  Scribe  did a great job supporting APA through the process of producing the XML for loading into Tizra, and we’d definitely recommend them to anyone interested in such a transition. It’s hard to overstate what a big step forward this is for Tizra as a platform and a company. XML has long been planned for in the product's architecture, but now for the first time, we have a working example that demonstrates t...

Princeton University Press Partners with Tizra to Take Einstein Papers Online

Unprecedented project will make nearly 30 volumes of Albert Einstein's papers available throughout the world. October 9, 2013 (Providence, RI) -- Princeton University Press has selected Tizra as the digital publishing platform it will use to make The Collected Papers of Albert Einstein available online.  One of the most ambitious publishing ventures ever undertaken in the documentation of the history of science, The Collected Papers will ultimately comprise more than 14,000 papers selected from all phases of Einstein's career and fill nearly 30 volumes. The online version hosted by Tizra will provide easy, searchable access to the full archive, and will offer features including: Powerful, bilingual search, with page-specific search results. Links between German text and corresponding English translations. Persistent, page-specific URLs to facilitate citation, referencing and discussion. Easy online viewing in all common web browsers, including those on the most p...

Scarcity Amid Abundance

One ramification of Chris Anderson's economics of abundance argument was nicely summarized by David Hornik: don't do one thing, do it all; don't sell one piece of content, sell it all; don't store one piece of data, store it all. The Economy of Abundance is about doing everything and throwing away the stuff that doesn't work. In the Economy of Abundance you can have it all But for most publishers this has been easier said than done. They may have an abundance of content, but building, feeding and tuning current online distribution and marketing systems is enough of a resource hog to dampen the experimental spirit at all but the richest. This, as you may have guessed, is the problem we're working on at Tizra. We think we're pretty close to solving it.