About

Posted Sat, 05/07/2008 - 20:22 by angch

This site is a place for us Byte Craft folks to dump, blog, rant about and host various stuff related to our work. Han, also known as angch, maintains this site. This is also site for us to experiment various things so there might be breakage from time to time. This site also suffers from shoemakers-kids-go-barefoot syndrome, sorry.

angch's picture

nginx https proxypass for php apps

Posted Tue, 09/02/2010 - 17:01 by angch

For various reasons (performance + clustering + slowloris protection), putting nginx in front of your apache+php application is a Good Thing. In addition to http, we usually let nginx fronting https for apache, so to simplify your load balanced php apps.

Some additional benefits of doing this:

  • This also enables you to serve multiple domains off a single IP (aka Named Based Virtual Hosts over https), something that Apache doesn't do very well. Your browser may complain loudly about mismatched https certs, though
  • HTTP referer hiding. If you have an internal custom site with less security, say http://private.home/ behind a firewall, you don't have to worry about users from the Internet accessing it, but any links from http://private.home/ to, say, http://www.bytecraft.com.my/ your browser will still leak the internal URL to the webmaster of the external site. Convert the internal site to https://private.home/ and all referers are not sent.

One downside is that the PHP app must realize that it is separated from the actual browser from itself. So the traditional way for checking the remote IP and whether a connection is https or not would not work. ($_SERVER['REMOTE_ADDR'] & $_SERVER['HTTPS'])

To fix this, there are two parts we must changed: get nginx to send additional headers, and get the PHP app to recognize the alternate headers.

1. nginx config

both http and https sections:

location / {
    proxy_pass http://10.0.0.1:80/; # Replace with your apache ip/port
    proxy_set_header Host $http_host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Scheme $scheme;
}
X-Forwarded-For and Scheme are the important ones here.

2. PHP

Highly dependant on your PHP app, but in general, search for $_SERVER['REMOTE_ADDR'] and $_SERVER['HTTPS'].

Example snippet to force https connection (say, for the login page):

if (isset($_SERVER['HTTP_SCHEME']) && $_SERVER['HTTP_SCHEME'] == 'http') {
    header('Location: https://'.$_SERVER['HTTP_HOST'] . $_SERVER['REQUEST_URI']);
    exit;
}

angch's picture

sugarcrm & memcache: Doing it Wrong

Posted Tue, 09/02/2010 - 12:10 by angch

Part one, tl, dr: memcache actually made sugarcrm slower.

Part two, tl, dr: Installing php5-mcrypt makes your sugarcrm site runs faster! Wish they'd note this more prominently.

We reinstalled sugarcrm the other day. Looked okay on development machine. Tested, everyone (for some values of "every") liked it. So we deployed it to the production machine. Used it for a while, then the dashboard and front page promptly slowed down to a crawl, taking up to 10 seconds to display when it was very snappy and loaded in sub-seconds.

After some false starts at troubleshooting, we backported the site back to the development machine. Slow. Okay, time to bring out the heavy duty tools and be serious about it.

1. Get Xdebug up

sudo apt-get install php5-xdebug     # If you haven't already
sudo vi /etc/php5/conf.d/xdebug.ini  # Or jed. jed is good.

Add following to xdebug.ini:

xdebug.profiler_enable=1
xdebug.profiler_output_dir=/tmp/xdebug

Set up the output directory, restart apache:

mkdir /tmp/xdebug
sudo chown www-data /tmp/xdebug
sudo invoked-rc.d apache2 restart

... and reload the offending slow sugarcrm page. xdebug spits out the diags at

/tmp/xdebug/cachegrind.out.13582

2. Cachegrind Analysis

The cachegrind file is a rather raw output, so we'd rather have a pretty GUI to visualize it. We'll use kcachegrind for this.

Rather than installing the GUI on the development server, we disabled xdebug, and copied the cachegrind output to our workstation. sudo apt-get install kcachegrind on our workstation will get us everything we need.

Start it up. (Applications / Programming / KCacheGrind). Load cachegrind.out.13582 that we copied over. Set the display to display absolute time (Right click on the data, "Show absolute cost"):

There. Out of the total runtime of 7.57 seconds (first row), 6.94 seconds is spent on php::Memcache->set. What?

Memcache making things slower? Wait, we didn't configure sugarcrm to use Memcache! Apparently sugarcrm automagically uses memcache if it detects it:

Sugar automatically enables external cache support once it detects a supported external cache method

3. Conclusion & Fix

It's memcache. It has to be. Time to test.

Before stopping memcached:

Server response time: 7.23 seconds.

After stopping memcached:

Q.E.D. More permanent fix:

sudo vi config_override.php
$GLOBALS['sugar_config']['external_cache_disabled_memcache'] = true;

I'm not sure if the problem is related to sugarcrm or the way we set up our server (all signs point to our version of memcache module for php), but this post is more about how we found the problem, and can be used for just about any php app.

P.S. sugarcrm 5.5a, Ubuntu 8.04 LTS (amd64), mysql 5.0.

P.P.S. Spoiler in title, I know.

Part 2: Speed, more speed!

Further testing still showed that our development server is still faster than production. More testing showed that the both development and production server called Sugar's PEAR's Blowfish routines(why sugarcrm is crypting so many things is beyond me) alot, but the development machine runs those faster. Hmmm.

Further looking at the source code, PEAR's blowfish has a Blowfish implementation in PHP as a fallback in case it can't load the mcrypt module for php. Installing php5-mcrypt and restarting the webserver fixed that.

angch's picture

subversion and https in Ubuntu Karmic

Posted Wed, 20/01/2010 - 17:29 by angch

Had a problem checking out an https subversion repository (self-signed) in Ubuntu Karmic, while Hardy is fine.

In short:

  • svn Hardy uses ra_dav for https
  • svn Karmic uses ra_neon (and libneon27-gnutls) for https.

gnutls is particularly sensitive about certs. So we need to work around it.

So do a:

sudo apt-get install libneon27
LD_PRELOAD=/usr/lib/libneon.so.27 /usr/bin/svn co https://myserver/myrepo
(use alias svn='LD_PRELOAD=/usr/lib/libneon.so.27 /usr/bin/svn')

Above fix comes from:

https://bugs.launchpad.net/ubuntu/+source/subversion/+bug/294648

kamal's picture

Drupal: Views block delta converted to md5 hash

Posted Fri, 08/01/2010 - 17:03 by kamal

Find out this only through reading the views.module code. In case of your views block display has a long name (> 32 chars)(the views name would be used as delta in block table), Views would hash it through md5. You can get the list of hashed names stored in the variable table as views_block_hashes. For example, mine:-

Array
(
    [ba843477361f490ca5c9e42548f0127d] => imagegallery_block_latest_images-block_1
)

The related code is in line 298 of views.module.

angch's picture

Ubuntu Server install requires PAE

Posted Wed, 09/12/2009 - 12:29 by angch

Testing a plain ubuntu 8.04.3 install on VirtualBox. Got the error

This kernel requires the following features not present on the CPU:
0:6
Unable to boot - please use a kernel appropriate for your CPU

Turned out 0:6 is PAE, and the -server series of kernels for Ubuntu requires it, rather than making use of it if available.

Don't need to downgrade your kernel to -generic, just turn on PAE in your machine or virtual machine. In the case of VirtualBox:

kamal's picture

Installing Drupal from command line

Posted Tue, 08/12/2009 - 19:04 by kamal

Using twill because until Drupal 7, there's no way you can install Drupal without opening the browser.

#debug http 1
go http://192.168.1.6:12000/install.php?profile=pulut&locale=en
code 200
find "To set up your Pulut database, enter the following information."
#show_cookies

fv 1 db_type pgsql
fv 1 db_path gmp
fv 1 db_user kamal
fv 1 db_pass abc123

submit op

find "All necessary changes to <em>./sites/default</em> and <em>./sites/default/settings.php</em> have been made. They have been set to read-only for security."

fv 1 site_mail kamal@bytecraft.com.my
fv 1 account[name] admin
fv 1 account[mail] kamal@bytecraft.com.my
fv 1 account[pass][pass1] abc123
fv 1 account[pass][pass2] abc123
fv 1 update_status_module[1] 0

submit op

find "Congratulations, Pulut has been successfully installed"

clear_cookies
go http://192.168.1.6:12000/

fv 1 name admin
fv 1 pass abc123

submit op
find "Log out"

It's pretty crude at the moment but I guess a good start. Couple with some other tools I'm working on, installing Drupal can be fully automated. The next step is to convert this twill script into Python equivalent to be more flexible.

Update

Here some good and better explanation on this from Lin Clark's blog.

kamal's picture

Drupal: Handling form field weight through CCK

Posted Tue, 01/12/2009 - 16:23 by kamal

If you have custom field added to the node form (probably using hook_form_alter), you have to manually set the #weight properties to order the element the way you want on the node form. CCK user definitely would miss the 'drag and drop' UI to rearrange CCK fields.

Turn out it possible to somehow 'register' your custom fields so that it can be rearranged using the CCK UI. Cool. I first found this in an old issue queue. Further 'googling' lead me to this blog post which describe it further.

The key is the hook_content_extra_fields that you can implement to return an array of fields to be managed by CCK. Then function content_extra_field_weight($node_type, $fieldname) can be use to set the weight properties on hook_view or hook_nodeapi view op.

helmi's picture

minify javascript using Google Closure Compiler

Posted Wed, 11/11/2009 - 12:26 by helmi

Optimize javascript using minify

"It combines multiple CSS or Javascript files, removes unnecessary whitespace and comments, and serves them with gzip encoding and optimal client-side cache headers."

Example of interesting urls:
* http://dev.bigproject/min/?f=static/map.js
* http://dev.bigproject/min/?b=static&f=map.js,taskpane.js,OpenLayers-2.8.js
* http://dev.bigproject/min/?g=js

Few challenges during setup
* 400 Bad Request -- check your path in config.php, especially $min_documentRoot

kamal's picture

Drupal urlencode: Dealing with ampersand in filename

Posted Mon, 09/11/2009 - 10:31 by kamal

A bug poping out with some files failed to download, returning 404. Looking closer at the issue, turned out the files that failed contained ampersand "&" in the filename. PHP would translate the "&" in request path as a key in $_GET variable. So something like:-

$ wget http://www.somesite.com/system/files/fm/somefile_with_&_ampersand.pdf

would result in PHP $_GET as:-

array(
  [q] => system/files/fm/somefile_with_
  [_ampersand] =>
)

urlencode'ing' the path is not enough because PHP would still translate that to an ampersand.

Syndicate content