This site is a place for us Byte Craft folks to dump, blog, rant about and host various stuff related to our work. Han, also known as angch, maintains this site. This is also site for us to experiment various things so there might be breakage from time to time. This site also suffers from shoemakers-kids-go-barefoot syndrome, sorry.
Ubuntu mirrors up and improved!
angch — Thu, 29/04/2010 - 12:01
The Malaysian to World internet connectivity was shot to heck recently, having more than 200ms (avg 300ms) ping times. This means our cronjobbed rsync based mirroring for ubuntu (archives and releases) was woefully slow, and the next cronjobbed rsync started before the previous one completes.
Spent sometime fixing that and got things back up to date, switched to ubumirror which at least has a lock to make sure we don't overlap mirroring.
The releases also required us to use a hybrid apache/nginx configuration to get some of the layout similar to the other official mirrors, otherwise you don't get listed on: https://launchpad.net/ubuntu/+cdmirrors .
All that done and tested, we're on a 3 hours release sync for the releases (getting you Lucid in record time) and 4 hours release sync for archives. Once things are a bit stabler, we'll run Push Mirroring.
tl/dr, Malaysians, get your CD images here: http://ubuntu.bytecraft.com.my/releases/
sources.list:
deb http://ubuntu.bytecraft.com.my/ubuntu/ YOUR_UBUNTU_VERSION_HERE main
deb-src http://ubuntu.bytecraft.com.my/ubuntu/ YOUR_UBUNTU_VERSION_HERE main Above is also https enabled if you can tolerate wrongly self-signed SSL cert.
P.S. mmu and upm also run mean, fast mirrors: https://launchpad.net/ubuntu/+mirror/archive.mmu.edu.my https://launchpad.net/ubuntu/+mirror/www.mirror.upm.edu.my-archive
P.P.S. OSCC has similar bandwidth with us: https://launchpad.net/ubuntu/+mirror/mirror.oscc.org.my-archive
Google-jstemplate: Iterate object with unknown properties
kamal — Wed, 21/04/2010 - 17:17
Iterating over list of object (array) is easy. Given the following template:-
<ul id="city">
<li jsselect="$this" jscontent="$this"></li>
</li>and our data (in json):-
var cities = [
"KL",
"KB",
"JB"
];
var processingContext = new JsEvalContext(cities);
var template = document.getElementById("frm-edit");
jstTemplate(processingContext, template);$this in the template refer to the object that we pass to JsEvalContext constructor.
nginx https proxypass for php apps
angch — Tue, 09/02/2010 - 17:01
For various reasons (performance + clustering + slowloris protection), putting nginx in front of your apache+php application is a Good Thing. In addition to http, we usually let nginx fronting https for apache, so to simplify your load balanced php apps.
Some additional benefits of doing this:
- This also enables you to serve multiple domains off a single IP (aka Named Based Virtual Hosts over https), something that Apache doesn't do very well. Your browser may complain loudly about mismatched https certs, though
- HTTP referer hiding. If you have an internal custom site with less security, say http://private.home/ behind a firewall, you don't have to worry about users from the Internet accessing it, but any links from http://private.home/ to, say, http://www.bytecraft.com.my/ your browser will still leak the internal URL to the webmaster of the external site. Convert the internal site to https://private.home/ and all referers are not sent.
One downside is that the PHP app must realize that it is separated from the actual browser from itself. So the traditional way for checking the remote IP and whether a connection is https or not would not work. ($_SERVER['REMOTE_ADDR'] & $_SERVER['HTTPS'])
To fix this, there are two parts we must changed: get nginx to send additional headers, and get the PHP app to recognize the alternate headers.
1. nginx config
both http and https sections:
location / {
proxy_pass http://10.0.0.1:80/; # Replace with your apache ip/port
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Scheme $scheme;
}2. PHP
Highly dependant on your PHP app, but in general, search for $_SERVER['REMOTE_ADDR'] and $_SERVER['HTTPS'].
Example snippet to force https connection (say, for the login page):
if (isset($_SERVER['HTTP_SCHEME']) && $_SERVER['HTTP_SCHEME'] == 'http') {
header('Location: https://'.$_SERVER['HTTP_HOST'] . $_SERVER['REQUEST_URI']);
exit;
}sugarcrm & memcache: Doing it Wrong
angch — Tue, 09/02/2010 - 12:10
Part one, tl, dr: memcache actually made sugarcrm slower.
Part two, tl, dr: Installing php5-mcrypt makes your sugarcrm site runs faster! Wish they'd note this more prominently.
We reinstalled sugarcrm the other day. Looked okay on development machine. Tested, everyone (for some values of "every") liked it. So we deployed it to the production machine. Used it for a while, then the dashboard and front page promptly slowed down to a crawl, taking up to 10 seconds to display when it was very snappy and loaded in sub-seconds.
After some false starts at troubleshooting, we backported the site back to the development machine. Slow. Okay, time to bring out the heavy duty tools and be serious about it.
1. Get Xdebug up
sudo apt-get install php5-xdebug # If you haven't already
sudo vi /etc/php5/conf.d/xdebug.ini # Or jed. jed is good.Add following to xdebug.ini:
xdebug.profiler_enable=1
xdebug.profiler_output_dir=/tmp/xdebugSet up the output directory, restart apache:
mkdir /tmp/xdebug
sudo chown www-data /tmp/xdebug
sudo invoked-rc.d apache2 restart... and reload the offending slow sugarcrm page. xdebug spits out the diags at
/tmp/xdebug/cachegrind.out.135822. Cachegrind Analysis
The cachegrind file is a rather raw output, so we'd rather have a pretty GUI to visualize it. We'll use kcachegrind for this.
Rather than installing the GUI on the development server, we disabled xdebug, and copied the cachegrind output to our workstation. sudo apt-get install kcachegrind on our workstation will get us everything we need.
Start it up. (Applications / Programming / KCacheGrind). Load cachegrind.out.13582 that we copied over. Set the display to display absolute time (Right click on the data, "Show absolute cost"):
There. Out of the total runtime of 7.57 seconds (first row), 6.94 seconds is spent on php::Memcache->set. What?
Memcache making things slower? Wait, we didn't configure sugarcrm to use Memcache! Apparently sugarcrm automagically uses memcache if it detects it:
Sugar automatically enables external cache support once it detects a supported external cache method
3. Conclusion & Fix
It's memcache. It has to be. Time to test.
Before stopping memcached:
Server response time: 7.23 seconds.After stopping memcached:
Q.E.D. More permanent fix:
sudo vi config_override.php$GLOBALS['sugar_config']['external_cache_disabled_memcache'] = true;I'm not sure if the problem is related to sugarcrm or the way we set up our server (all signs point to our version of memcache module for php), but this post is more about how we found the problem, and can be used for just about any php app.
P.S. sugarcrm 5.5a, Ubuntu 8.04 LTS (amd64), mysql 5.0.
P.P.S. Spoiler in title, I know.
Part 2: Speed, more speed!
Further testing still showed that our development server is still faster than production. More testing showed that the both development and production server called Sugar's PEAR's Blowfish routines(why sugarcrm is crypting so many things is beyond me) alot, but the development machine runs those faster. Hmmm.
Further looking at the source code, PEAR's blowfish has a Blowfish implementation in PHP as a fallback in case it can't load the mcrypt module for php. Installing php5-mcrypt and restarting the webserver fixed that.
subversion and https in Ubuntu Karmic
angch — Wed, 20/01/2010 - 17:29
Had a problem checking out an https subversion repository (self-signed) in Ubuntu Karmic, while Hardy is fine.
In short:
- svn Hardy uses ra_dav for https
- svn Karmic uses ra_neon (and libneon27-gnutls) for https.
gnutls is particularly sensitive about certs. So we need to work around it.
So do a:
sudo apt-get install libneon27
LD_PRELOAD=/usr/lib/libneon.so.27 /usr/bin/svn co https://myserver/myrepoAbove fix comes from:
https://bugs.launchpad.net/ubuntu/+source/subversion/+bug/294648
Drupal: Views block delta converted to md5 hash
kamal — Fri, 08/01/2010 - 17:03
Find out this only through reading the views.module code. In case of your views block display has a long name (> 32 chars)(the views name would be used as delta in block table), Views would hash it through md5. You can get the list of hashed names stored in the variable table as views_block_hashes. For example, mine:-
Array
(
[ba843477361f490ca5c9e42548f0127d] => imagegallery_block_latest_images-block_1
)
The related code is in line 298 of views.module.
Ubuntu Server install requires PAE
angch — Wed, 09/12/2009 - 12:29
Testing a plain ubuntu 8.04.3 install on VirtualBox. Got the error
This kernel requires the following features not present on the CPU:
0:6
Unable to boot - please use a kernel appropriate for your CPUTurned out 0:6 is PAE, and the -server series of kernels for Ubuntu requires it, rather than making use of it if available.
Don't need to downgrade your kernel to -generic, just turn on PAE in your machine or virtual machine. In the case of VirtualBox:
Installing Drupal from command line
kamal — Tue, 08/12/2009 - 19:04
Using twill because until Drupal 7, there's no way you can install Drupal without opening the browser.
#debug http 1
go http://192.168.1.6:12000/install.php?profile=pulut&locale=en
code 200
find "To set up your Pulut database, enter the following information."
#show_cookies
fv 1 db_type pgsql
fv 1 db_path gmp
fv 1 db_user kamal
fv 1 db_pass abc123
submit op
find "All necessary changes to <em>./sites/default</em> and <em>./sites/default/settings.php</em> have been made. They have been set to read-only for security."
fv 1 site_mail kamal@bytecraft.com.my
fv 1 account[name] admin
fv 1 account[mail] kamal@bytecraft.com.my
fv 1 account[pass][pass1] abc123
fv 1 account[pass][pass2] abc123
fv 1 update_status_module1 0
submit op
find "Congratulations, Pulut has been successfully installed"
clear_cookies
go http://192.168.1.6:12000/
fv 1 name admin
fv 1 pass abc123
submit op
find "Log out"It's pretty crude at the moment but I guess a good start. Couple with some other tools I'm working on, installing Drupal can be fully automated. The next step is to convert this twill script into Python equivalent to be more flexible.
Update
Here some good and better explanation on this from Lin Clark's blog.
Drupal: Handling form field weight through CCK
kamal — Tue, 01/12/2009 - 16:23
If you have custom field added to the node form (probably using hook_form_alter), you have to manually set the #weight properties to order the element the way you want on the node form. CCK user definitely would miss the 'drag and drop' UI to rearrange CCK fields.
Turn out it possible to somehow 'register' your custom fields so that it can be rearranged using the CCK UI. Cool. I first found this in an old issue queue. Further 'googling' lead me to this blog post which describe it further.
The key is the hook_content_extra_fields that you can implement to return an array of fields to be managed by CCK. Then function content_extra_field_weight($node_type, $fieldname) can be use to set the weight properties on hook_view or hook_nodeapi view op.
