angch's blog
Ubuntu mirrors up and improved!
angch — Thu, 29/04/2010 - 12:01
The Malaysian to World internet connectivity was shot to heck recently, having more than 200ms (avg 300ms) ping times. This means our cronjobbed rsync based mirroring for ubuntu (archives and releases) was woefully slow, and the next cronjobbed rsync started before the previous one completes.
Spent sometime fixing that and got things back up to date, switched to ubumirror which at least has a lock to make sure we don't overlap mirroring.
The releases also required us to use a hybrid apache/nginx configuration to get some of the layout similar to the other official mirrors, otherwise you don't get listed on: https://launchpad.net/ubuntu/+cdmirrors .
All that done and tested, we're on a 3 hours release sync for the releases (getting you Lucid in record time) and 4 hours release sync for archives. Once things are a bit stabler, we'll run Push Mirroring.
tl/dr, Malaysians, get your CD images here: http://ubuntu.bytecraft.com.my/releases/
sources.list:
deb http://ubuntu.bytecraft.com.my/ubuntu/ YOUR_UBUNTU_VERSION_HERE main
deb-src http://ubuntu.bytecraft.com.my/ubuntu/ YOUR_UBUNTU_VERSION_HERE main Above is also https enabled if you can tolerate wrongly self-signed SSL cert.
P.S. mmu and upm also run mean, fast mirrors: https://launchpad.net/ubuntu/+mirror/archive.mmu.edu.my https://launchpad.net/ubuntu/+mirror/www.mirror.upm.edu.my-archive
P.P.S. OSCC has similar bandwidth with us: https://launchpad.net/ubuntu/+mirror/mirror.oscc.org.my-archive
nginx https proxypass for php apps
angch — Tue, 09/02/2010 - 17:01
For various reasons (performance + clustering + slowloris protection), putting nginx in front of your apache+php application is a Good Thing. In addition to http, we usually let nginx fronting https for apache, so to simplify your load balanced php apps.
Some additional benefits of doing this:
- This also enables you to serve multiple domains off a single IP (aka Named Based Virtual Hosts over https), something that Apache doesn't do very well. Your browser may complain loudly about mismatched https certs, though
- HTTP referer hiding. If you have an internal custom site with less security, say http://private.home/ behind a firewall, you don't have to worry about users from the Internet accessing it, but any links from http://private.home/ to, say, http://www.bytecraft.com.my/ your browser will still leak the internal URL to the webmaster of the external site. Convert the internal site to https://private.home/ and all referers are not sent.
One downside is that the PHP app must realize that it is separated from the actual browser from itself. So the traditional way for checking the remote IP and whether a connection is https or not would not work. ($_SERVER['REMOTE_ADDR'] & $_SERVER['HTTPS'])
To fix this, there are two parts we must changed: get nginx to send additional headers, and get the PHP app to recognize the alternate headers.
1. nginx config
both http and https sections:
location / {
proxy_pass http://10.0.0.1:80/; # Replace with your apache ip/port
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Scheme $scheme;
}2. PHP
Highly dependant on your PHP app, but in general, search for $_SERVER['REMOTE_ADDR'] and $_SERVER['HTTPS'].
Example snippet to force https connection (say, for the login page):
if (isset($_SERVER['HTTP_SCHEME']) && $_SERVER['HTTP_SCHEME'] == 'http') {
header('Location: https://'.$_SERVER['HTTP_HOST'] . $_SERVER['REQUEST_URI']);
exit;
}sugarcrm & memcache: Doing it Wrong
angch — Tue, 09/02/2010 - 12:10
Part one, tl, dr: memcache actually made sugarcrm slower.
Part two, tl, dr: Installing php5-mcrypt makes your sugarcrm site runs faster! Wish they'd note this more prominently.
We reinstalled sugarcrm the other day. Looked okay on development machine. Tested, everyone (for some values of "every") liked it. So we deployed it to the production machine. Used it for a while, then the dashboard and front page promptly slowed down to a crawl, taking up to 10 seconds to display when it was very snappy and loaded in sub-seconds.
After some false starts at troubleshooting, we backported the site back to the development machine. Slow. Okay, time to bring out the heavy duty tools and be serious about it.
1. Get Xdebug up
sudo apt-get install php5-xdebug # If you haven't already
sudo vi /etc/php5/conf.d/xdebug.ini # Or jed. jed is good.Add following to xdebug.ini:
xdebug.profiler_enable=1
xdebug.profiler_output_dir=/tmp/xdebugSet up the output directory, restart apache:
mkdir /tmp/xdebug
sudo chown www-data /tmp/xdebug
sudo invoked-rc.d apache2 restart... and reload the offending slow sugarcrm page. xdebug spits out the diags at
/tmp/xdebug/cachegrind.out.135822. Cachegrind Analysis
The cachegrind file is a rather raw output, so we'd rather have a pretty GUI to visualize it. We'll use kcachegrind for this.
Rather than installing the GUI on the development server, we disabled xdebug, and copied the cachegrind output to our workstation. sudo apt-get install kcachegrind on our workstation will get us everything we need.
Start it up. (Applications / Programming / KCacheGrind). Load cachegrind.out.13582 that we copied over. Set the display to display absolute time (Right click on the data, "Show absolute cost"):
There. Out of the total runtime of 7.57 seconds (first row), 6.94 seconds is spent on php::Memcache->set. What?
Memcache making things slower? Wait, we didn't configure sugarcrm to use Memcache! Apparently sugarcrm automagically uses memcache if it detects it:
Sugar automatically enables external cache support once it detects a supported external cache method
3. Conclusion & Fix
It's memcache. It has to be. Time to test.
Before stopping memcached:
Server response time: 7.23 seconds.After stopping memcached:
Q.E.D. More permanent fix:
sudo vi config_override.php$GLOBALS['sugar_config']['external_cache_disabled_memcache'] = true;I'm not sure if the problem is related to sugarcrm or the way we set up our server (all signs point to our version of memcache module for php), but this post is more about how we found the problem, and can be used for just about any php app.
P.S. sugarcrm 5.5a, Ubuntu 8.04 LTS (amd64), mysql 5.0.
P.P.S. Spoiler in title, I know.
Part 2: Speed, more speed!
Further testing still showed that our development server is still faster than production. More testing showed that the both development and production server called Sugar's PEAR's Blowfish routines(why sugarcrm is crypting so many things is beyond me) alot, but the development machine runs those faster. Hmmm.
Further looking at the source code, PEAR's blowfish has a Blowfish implementation in PHP as a fallback in case it can't load the mcrypt module for php. Installing php5-mcrypt and restarting the webserver fixed that.
subversion and https in Ubuntu Karmic
angch — Wed, 20/01/2010 - 17:29
Had a problem checking out an https subversion repository (self-signed) in Ubuntu Karmic, while Hardy is fine.
In short:
- svn Hardy uses ra_dav for https
- svn Karmic uses ra_neon (and libneon27-gnutls) for https.
gnutls is particularly sensitive about certs. So we need to work around it.
So do a:
sudo apt-get install libneon27
LD_PRELOAD=/usr/lib/libneon.so.27 /usr/bin/svn co https://myserver/myrepoAbove fix comes from:
https://bugs.launchpad.net/ubuntu/+source/subversion/+bug/294648
Ubuntu Server install requires PAE
angch — Wed, 09/12/2009 - 12:29
Testing a plain ubuntu 8.04.3 install on VirtualBox. Got the error
This kernel requires the following features not present on the CPU:
0:6
Unable to boot - please use a kernel appropriate for your CPUTurned out 0:6 is PAE, and the -server series of kernels for Ubuntu requires it, rather than making use of it if available.
Don't need to downgrade your kernel to -generic, just turn on PAE in your machine or virtual machine. In the case of VirtualBox:
Postgis recipe: Extracting MULTILINESTRING into LINESTRING
angch — Wed, 30/09/2009 - 12:35
insert into linestringtable (the_geom, col1, col2, col3)
select st_geometryn(the_geom,n), col1, col2, col3
from
multilinestringtable,
generate_series(1, (select max(NumGeometries(the_geom)) from multilinestringtable)) n
where st_geometryn(the_geom,n) is not null
and GeometryType(st_geometryn(the_geom,n)) ='LINESTRING';Not horribly efficient, but gets the job done.
nginx: turn off sendfile when serving content through nfs
angch — Tue, 08/09/2009 - 15:10
Protip: turn off sendfile in nginx if you're serving content via nfs.
You take a performance hit, but it's stabler overall.
Symptom: You use nfs, and nginx takes 100% CPU, nothing gets served, and dmesg or /var/log/messages contains:
... comm: nginx Not tainted...
... [do_sendfile....
... [sys_sendfile64+.....Malaysian state boundaries in SVG
angch — Thu, 06/08/2009 - 20:32
Malaysian state boundaries, in SVG compressed format as requested by Seah Hong Yee.
I made the .svg, converted from an internal database, but I'm not certain of the copyright of the original information. So I can't really put this under Creative Commons, sorry. But I hope it's okay under fair use.
Enjoy.
Revisiting Linux swap space size calculations
angch — Wed, 29/07/2009 - 21:33
Back in the early days of Linux, the rule of thumb for sizing your swap partition is "twice the amount of RAM". But does that still apply to modern hardware where we regularly have 16 GB or more?
4 schools of thought:
- $ram * 2 (I call it the "jurassic" method)
- $ram > 2048 ? $ram + 2048 : $ram * 2 (aka "redhat", see below)
- $ram <= 2 GB ? 1.5 * $ram : ($ram <= 8 GB ? $ram : 0.75 * $ram) (aka "oracle", see below)
- 4GB (I call it "angch's")
Last school of thought is mine, and the rationale behind it includes:
Mapnik 0.6.0 Ubuntu Hardy 64bit packages
angch — Wed, 17/06/2009 - 12:05
Backported mapnik 0.6.0 packages (based on Debian unstable's) to Ubuntu Hardy amd64 attached below:
