When you use a server image to create a VPS not everything is the latest or best version. So here's the steps for updating PHP to the latest release, which at this time is versio 5.4. While you could this manually adding in the repositories available from the community is easier to maintain.

wget http://dl.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm
wget http://rpms.famillecollet.com/enterprise/remi-release-5.rpm
rpm -Uvh remi-release-5*.rpm epel-release-5*.rpm

EPEL
Extra Packages for Enterprise Linux (EPEL) is a
volunteer-based community effort from the Fedora project to create a repository of high-quality add-on packages for Red Hat Enterprise (RHEL) and its compatible spin-offs such as CentOS or Scientific Linux. Fedora is the upstream of RHEL and add-on packages for EPEL are sourced from the Fedora repository primarily and built against RHEL.

REMI
This is a repository created by Remi Collet with the aim to give support for old Fedora releases to allow them to install recent software, also it gives you the ability to have your CentOS/RHEL systems with the latest software available in the net.

Update PHP to the latest one in the repository

yum --enablerepo=remi update php php-*
/etc/init.d/httpd restart

I always forget that in a raw CentOS server install the ip table needs to be configured to allow http calls on port 80 and port 10000 if you want to use Webmin. If you don't do this then the web server will not respond to browser calls because it can't.

Rules created with the iptables command are stored in memory. If the system is restarted before saving the iptables rule set, all rules are lost. For netfilter rules to persist through system reboot, they need to be saved.

# iptables -I INPUT 5 -m tcp -p tcp --dport 80 -j ACCEPT
# iptables -I INPUT 5 -m tcp -p tcp --dport 10000 -j ACCEPT

# sudo service iptables save

# service iptables restart