You might be wondering  why I am saying all these here. Well, I suspected that the problem was actually because of my SQL query for delta load.  But to be sure, I wanted to see the query being executed by Solr DataImportHandler. As always I turned to Google for assistance, and I finally reached the MySQL documentation on the General Query Log. Voila! This was exactly what I wanted. All I had to do was use the –log=[filename] parameter and all my queries will be logged to the specified log file. Nice, isn’t it?

Now I have to stop my running MySQL server and restart it with the –log switch, in addition to the other regular options. But there was a problem, I was not sure of the other required parameters. You can use the ps utility, when the MySQL server is running, to find out the normal parameters.

ps -ax | grep mysql

For me the output was

/usr/local/mysql/bin/mysqld –basedir=/usr/local/mysql –datadir=/usr/local/mysql/data –user=mysql –pid-file=/usr/local/mysql/data/localhost.pid –port=3306 –socket=/tmp/mysql.sock

Now shutdown the MySQL server.

// On Mac
/Library/StartupItems/MySQLCOM/MySQLCOM stop
// For other Linux/Unix variants try
/etc/init.d/mysqld restart
service mysql restart

Start mysqld with –log option

/usr/local/mysql/bin/mysqld --basedir=/usr/local/mysql --datadir=/usr/local/mysql/data --user=mysql --pid-file=/usr/local/mysql/data/localhost.pid --port=3306 --socket=/tmp/mysql.sock --log=/tmp/query.log

The general query log contains lots of irrelevant information. To view the log after filtering out the unwanted details use tail and grep as given below

tail -f /tmp/query.log | grep -v Connect | grep -v Quit

The amount of information added to the file is quite large. If you are using this on a production server, I recommend turning off the logging once you are done.

awk '{if (NR == 1)print >> "header.txt"; else print >> "body.txt";}' input.txt
Where NR is a built-in variable that contains the number of the current record / line,
input.txt is the input file,
header.txt is output file for header,
body.txt is the output file for the remaining content

Awk reads the input file (input.txt) line by line and checks whether the current line is the first line. If so the content is appended ( >> ) to the file header.txt. Else the content is appended to body.txt.

Everything I have mentioned so far might be known to you already. So why put a false title? Well, I am not finished yet. Yes, you can get a free every minute uptime check with free sms notification when your website is down, depending on your location and mobile service provider.

Pingdom is one of the best uptime monitoring services available. I had been their premium subscriber till last month. When my subscription period was about to end, I started looking for an alternative, since I felt I was not using my subscription fully. I was using only one check, and a premium account allowed up to 5 checks. Most of the free service I checked were limited in frequency of checks (max being every 5 mins). Finally I came across Pingdom’s twitter account and found that they were offering free accounts with one every min check. There was also 20 free sms credits available with the account. What happens once that is over? Their alert methods also includes sending direct messages to your Twitter account. Now twitter allows sms updates for tweets from people you follow. By setting up Direct Message alert in Pingdom and following PingdomAlert on Twitter with sms alerts turned on, you can get unlimited free sms alerts when your website is down.

Disclaimer:
When you are depending on two services, it is possible that problem with any of providers can cause delay’s and reduce the overall effectiveness of the system. Even if Pingdom sends the alert within a minite, there is no guarantee that twitter will send the sms alert on time. If you have a critical application, do not rely on such an unreliable setup and go for a premium subscription. When I checked, I got sms notifications within 1 min of receiving a Direct Message. Also, sms check is not much important for me since I have my mailbox open most of the time, and usually detect downtimes within 5 mins from email alerts. Your case might be different, so use at your own risk.

Enabling gzip compression in an existing PHP file is very simple. Add the following line at the beginning of the file (it should appear before any output statement).

<?php
ob_start ("ob_gzhandler");
?>

Very simple, right? But what if we have hundreds of files to edit? Don’t worry, there is solution for that too. We can use the PHP configuration variable auto_prepend_file to automatically include the above code at the beginning of all PHP files. Copy the above code to a file named prepend.php and place it at the root of your website (~/public_html in CPanel, ~/httpdocs in Plesk).

Now we are going to automatically include the above file at the beginning of all PHP files using auto_prepend_file. Depending on how PHP is configured on your server, the steps for modifying the PHP configuration variable is also different.

If PHP is configured in CGI/FastCGI mode (HostGator uses this mode), we will be using php.ini file to make the above change. Create a file named php.ini at the root of your website and copy the following code into it. If the file already exists, append it to the end of the file.

auto_prepend_file=<full-path-to-document-root>/prepend.php

If PHP is loaded as apache module (mod_php), we will have to use .htaccess to achieve the same effect. Create a file named .htaccess at the web root and copy the following code into it. If the file exists, append at the end of the file.

php_value auto_prepend_file <full-path-to-document-root>/prepend.php

In both methods, replace <full-path-to-document-root> with the full path to your website root directory, for example /home/joyce/public_html.

Now the prepend.php file is automatically included by the PHP interpreter, whenever a PHP file is requested. But this method (as of now) does not work for non PHP files like HTML, CSS, JavaScript etc, which should also be compressed. This is because these files are handled directly by Apache and is not passed to the PHP interpreter. To ensure that these files are also compressed, we have to instruct Apache to pass these files to the PHP interpreter before sending them to the browser. For this, lets once again go back to .htaccess. Append the following code at the end of your .htaccess file (create one, if you don’t have  it already).

<FilesMatch "\.(htm|html|css|js)">
	ForceType application/x-httpd-php
</FilesMatch>

The above code instructs Apache that files ending with .js, .htm, .html, .css and .js extensions are also PHP files and should be passed through PHP interpreter. The only problem remaining is that Content-type header for all files have been changed to text/html. To fix this we need to check the requested file-name and set the correct Content-type header depending on the file extension. In order to do this, open prepend.php and replace the current code with

<?php
// Start output buffering
ob_start ("ob_gzhandler");

// Set the correct content type header depending on filename
$arContentType = array('htm' => 'text/html', 'html' => 'text/html', 'css' => 'text/css', 'js' => 'text/javascript');
if(strpos($_SERVER['REQUEST_URI'], '.') !== FALSE){
	$ext = explode('.', $_SERVER['REQUEST_URI']);
	$ext = array_pop($ext);
	if(($pos = strpos($ext, '?')) !== FALSE){
		$ext = substr($ext, 0, $pos);
	}
	if(isset($arContentType[$ext])){
		header ("Content-type: {$arContentType[$ext]}; charset: UTF-8");
	}
}

// You can also set expiration headers in this file
?>

That’s all Folks. You have successfully enabled Gzip compression on your server without using any Apache modules.

This is a working code. But I have tested it only with mod_php configuration. You have to be very careful when enabling PHP code in JS/CSS files. There may be many flaws in this code. If you find any, please let me know. We can further improve this code by adding expiration and cache headers to enable client side caching.

It looks like Apple has finally decided to fight back by entering Google’s arena with the acquisition of the mobile ad company Quattro. Google had recently acquired AdMob, a four year old ad network, which had risen to be one of the most important players in the Mobile Ad industry. The deal is under scrutiny by the U.S. regulators. There were rumors that Apple also wanted to buy AdMob, but was out-bidden by Google.

The acquisition also puts Apple in direct competition with other companies like Microsoft and Yahoo.

Technical specifications of the phone can be found here. The device designed by Google and built by HTC is powered by a 1GHz Snapdragon processor and 512MB of ROM & 512MB of RAM. It includes features like GPS, accelerometer, proximity sensor, a 5 mega pix camera with LED flash and an additional microphone for noise cancellation. With a weight of 130 gms and a thickness of 11.5mm, Google boasted that the handset is no thicker than a number two pencil, and no heavier than a Swiss Army knife.

Unfortunately, the phone is not available in India. I see the following message when visiting the official webpage

Sorry, the Nexus One phone is not available in your country.

According to the sales page of the phone, Nexus One is currently available only in the U.S. The phone is available for a contract free price of $529 or a subsidized rate of $179 on a two year contract with T-Mobile. The page says that the Google phone will be available in US from Verizon wireless and in Europe from Vodafone in spring 2010.

The phone can be personalized with a two line engraving at the back of the phone.

Hopefully, after almost a month of anticipation, the wait is finally getting over. Google has called for an “Android Press Gathering” tomorrow. Everyone believe that the new phone will be announced at the press meet.

Encouraged by Google’s regular practice, I checked the page www.google.com/phone and found that the page is different from the standard Google 404 page. Which suggests that there is something going on there. So keep checking the url as it may be official page of the Google phone, once it is released.

Update: Engadget has posted a review of Nexus One.

wget http://www.joycebabu.com/my-photo.jpg
Where http://www.joycebabu.com/my-photo.jpg is the file to be downloaded.

By default wget saves the file with the same name as the fetched file. For example, the above command will save the file as my-photo.jpg in the current working directory. If a file already exists with the given name, the downloaded file will be named as my-photo.jpg.1, my-photo.jpg.2 etc until a non existent filename is found.

It is possible to explicitly specify a different name for the downloaded file. This is possible using the -O switch (--output-document also works, but I believe short is sweet). The new command is

wget -O photo.jpg http://www.joycebabu.com/my-photo.jpg
Where photo.jpg is the new filename.

But be careful while using the -O switch. If there is an existing file with the same name, it will be overwritten with the new file.

I have been using the Secure Copy (scp) utility for copying files between my local server and development server. Sometimes I have to copy more than one file. Previously I used to copy the files one at a time. This is very annoying, as you have to type the password every time you use the command . But it is possible to copy multiple files using scp, just like the copy (cp) utility.

When you have to copy multiple files to your remote server, the syntax is similar to the cp command.

scp file1.sql file2.sh joyce@joycebabu.com:~/upload

Where file1.sql and file2.sh are the files to be copied, joyce is the username, joycebabu.com is the hostname and ~/upload is the destination directory on the remote server.

In order to download multiple files from the remote server, the command to be used is

scp joyce@joycebabu.com:"file1.log file2.log" ~/logs

Where file1.log and file2.log are the files to be downloaded and ~/logs is the destination directory on the local server. Notice the quotes around the filenames. This ensures that the filenames list is not parsed by the local shell and is passed to the remote shell. Similarly, when you want to download files using wildcards (*.php, files_?.log etc), you should enclose the name within quotes to ensure that the expansion is done by the remote server.

The -r option can be used to copy directories recursively.

scp -r joyce@joycebabu.com:~/logs ~/logs

This may not be a lifesaver tip and the time gained by this method may be small. After all, when a large number of files are to be transferred, I use FTP or tar my files and copy it. But at times when things go wrong, even this small gain can help.

As usual, I approached Google for a solution and it introduced me to two different solutions – AWK (a programming language) and Sed (a unix utility). There is only a very slight difference between the two commands.

awk ‘/– Table structure for table .tbl_first./,/– Table structure for table .tbl_second./{print}’ mydumpfile.sql > mytable.sql

sed -ne ‘/– Table structure for table .tbl_first./,/– Table structure for table .tbl_second./p’ mydumpfile.sql > mytable.sql

Here tbl_first is the table I wanted to extract and tbl_second was the table below that. The above commands will search the file mydumpfile.sql and extract the text between the start string (– Table structure for table .tbl_first.) and end string (– Table structure for table .tbl_second.). The dots before and after the table name are wildcard character to match the engrave character, which has a special meaning in shell commands. The {print} option (p in sed) prints the extracted string, which is then redirected to the file mytable.sql.

But that didn’t solve my problem completely. I was not sure of the order of tables in the .sql file. This time  grep (another powerful unix utility) came to my rescue. The following command lists all the tables in the file mydumpfile.sql in the same order in which they appear in the file.

grep ‘Table structure’ mydumpfile.sql | cut -d’`’ -f2

I don’t know a lot about shell commands. But with my very limited experience I can say that they are extremely powerful. Two small lines of code saved me a lot of time.