@pacharest: New cluster to be deployed very soon. #VMWARE & #SMART_NETWORK (#PUPPET #DRBD #ZABBIX #SNORT)
Couples of stats/facts.
As I look over 6 very interesting projects overview on my desk, I’m forced to do a bit of thinking about how the last year went by. A year is a lot of time, and so much plans finally came to fruition that I can’t think of listing them all here today. Which is kinda a good sign for me and my enterprise ;-)
Most of my readers doesn’t really know who I am, even when you take into account that I blog under my real name. Most don’t know that I bought a condo in Hull (now part of Gatineau, near Ottawa – the capital of Canada), that I still have a rent in Montreal, that I proposed to my girlfriend (she said “Yes!”), that I own a dog (greatest experience of forcing a regular schedule I ever had), that my greatest motivation in life is to be able to go where I want, whenever I want. My dream is going back to Yosemite, California… and bring hiking gear.
Another big aspect of my life is my business, Les Laboratoires Phoenix. I’ve been working full time at it for the last 9 months and its been a great experience. Over those months : I’ve worked with clients from 7 countries, contributed to 3 major open source projects, went to the “Free Software Foundation” Libre Planet confrence in Boston, went to the DefCon in Las Vegas, I’ve been named SME for {Zabbix, Zimbra, Asterisk, OpenLDAP, extended LAMP Stack, Mailman, GlusterFS, Lustre, MySQL, Cloud Computing, …}, 3 of my articles have been published (>40K prints), and I’m involved in a book project (from a major publisher)…
And, even thinking about all those achievements, I still look for the future of Les Laboratoires Phoenix. I guess that working with startups influenced me a lot : those 6 projects are all different from each others, they represent good revenue potential (clear business plan) and require low capital input to be started. So, I guess I’ll stop speaking about them and work ;-). Btw, two of those projects would be online services (SAAS) for well known parts of Internet infrastructure (not webserver). Another is a cloud computing infrastructure services based in Montreal (this one if almost finished! & I got an hardware provider)… A lot of fun to be had.
More news to come.
Swekey – An authentication gizmo for Windows, Mac OSX, GNU/Linux
Through my connection with PraizedMedia (a client of Les Laboratoires Phoenix- managed data infrastructure), I received a ‘Swekey‘ device. It look like an normal USB key, but their website seem to push toward something much more useful (and potentially dangerous). Hence, I decided to try it. It is advertised as :
The swekey is a small USB key that secures access to any swekey enabled web sites.
Swekey secured web sites won’t let you login without your swekey plugged to your computer.
The swekey can also be used to secure corporate’s intranet, unix servers access, and database administration.
[...]

Swekey device, Photo by Pascal Charest
The website mention integration with WordPress, SSH, putty, MediaWiki, Zabbix, Magento, SugarCRM… and much more… In fact they even speak about integration with any OpenID enabled websites – Might be very cool and interesting. Lets see how it work.
I’m an hacker at heart, so I don’t normally read much of a device documentation, but in this case – I was lost. How is the device working? Is it a key with auto-run partition + dedicated browser, is it the equivalent of an RSA key, is there any software to install ? To answer my questions, what would be better that some tests in a protected GNU/Linux workstation (which is what normal people do : plug it in and see what happen):
The device auto-detection work and recognize the device as an USB CDROM drive (from dmesg):
usb 2-8: new full speed USB device using ohci_hcd and address 3
usb 2-8: configuration #1 chosen from 1 choice
Initializing USB Mass Storage driver…
scsi10 : SCSI emulation for USB Mass Storage devices
usbcore: registered new interface driver usb-storage
USB Mass Storage support registered.
usb-storage: device found at 3
usb-storage: waiting for device to settle before scanning
usb-storage: device scan complete
scsi 10:0:0:0: CD-ROM Musbe Swekey 1.03 PQ: 0 ANSI: 0
sr1: scsi-1 drive
sr 10:0:0:0: Attached scsi CD-ROM sr1
sr 10:0:0:0: Attached scsi generic sg3 type 5
cdrom: This disc doesn’t have any tracks I recognize!
usb 2-8: reset full speed USB device using ohci_hcd and address 3
Then : Nothing. No auto-mount, no dialog box… Kinda of left there. The partition cannot be mounted…
Going to their website, I learn the official working steps: “BUY” (pseudo-done), “PLUG” (done), “REGISTER” (ugh?) and I’m “READY”. The REGISTER (the step I’m at, right ?) section give me an error of ‘missing plug-in’ from Mozilla Firefox 3.0.14. Ok, browsing “Support”/”Download” inform me of missing dependencies (a software must be installed) to access the device. I download the x64 GNU/Linux version and … hum ?
pcharest@hydra:~/Desktop/swekey$ cat README
Swekey client
This package install:
- the swekey-client command line tool
- the swekey HAL module
- the swekey Mozilla pluginThe swekey-client command line tool gives you the list of plugged swekeys
and let you calculate OTPs with them.type:
swekey-client –help
to get the available optionsTo install swekey-client just type:
sudo ./install
or
./install
if you are rootTo uninstall swekey-client just type:
sudo ./uninstall
or
./uninstall
if you are root
I have no idea what is an OTP but let say I try installing the client:
sudo ./install
and validate the device is detected:
./swekey-client –list
It work and give me a device ID. Good, at least the device is known by the system. I still don’t know how it should work. I guess I should be installing the Mozilla plug-in the readme mentionned, but… I never found it. I guess the client install worked (and it was included) because after a Mozilla reload, the Manage section of their web page give (or might also be one of the random file I clicked on) :
Registration is not mandatory but it will allow you to disable a lost or stolen Swekey.
So… I don’t really need to register the key… lets try it then (which I’ve been trying to do for quite a long time at this point).
I own quite a few Zabbix servers, so, from the list of supported service :
ZABBIX is an enterprise-class open source distributed monitoring solution.
A swekey integration exists, it is still a patch but you can ask for it if you need to test it.
Ok, still want to test the device – So i try with MediaWiki:
And it started to work well : creation of an account (user+password), then I get asked if I want to bind this account to my Swekey. This won’t allow me to auto-login but will require the key to be present in any computer (with the installed software) to access the account.
Summary: As a summary, I’d say that while it give a boosted security (require the Swekey to log) – it does seem to go a bit over the limit of the permanent fight between conviviality and security. Installing the software is complicated and might be very problematic on system without administrator access… Personally, having tried both, I would prefer Paypal key ID to be integrated to more website. There is no need to ‘install’ the software on any computer and it give you the same added security the Swekey does.
top sysadmin stuff
Being challenged everyday to augment my productivity, here is a few quick tricks/software helping system administrator.
1. BlackBerry
Yeah, I know. It was an easy one – and easy to expect since I’ve bought a Storm. Employees get to hate them (since they are always hooked to the business) but as owner of a small business, I NEED to be informed of everything going on. The ability of answering my email / instant messanging while in route between Montreal & Ottawa is of prime importance. My clients doesn’t need to know where I am or what I am doing, they know I’m ready to help them.
The BlackBerry by itself is not as feature-complete as the iPhone seem to be. Using the pre-loaded email client with gmail just doesn’t cut it. It’s using IMAP and discarding all your filter/labels for incoming messages. There is an alternative : gmail mobile application. Available from the central mobile application repository of Google. Using a customized alert setting, you can be informed when you have new mail (in your inbox), while preserving your filter/label configuration. While you are there, you should also install the maps application, can always be handy.
Another “must-have” app. for sysadmin is MidpSSH. Which, as its name make it pretty clear, is a SSH/Telnet client. There have been a few reports of incompatibility between Storm and midpssh – yet, with an up-to-date OS/taking into account that your device often capitalize the first letter (of a username)/openssh is case-sensitive, you should not have any problems to connect to GNU/Linux systems.
2. Monitoring software
A good monitoring system watching over your network is a life saver and all the difference between you informing your client of a system failure or the other way around. Nagios is pretty well known and getting help is very easy since the community is so dynamic. Another software doing the same job is Zabbix. I do have some predisposition toward it, being a certified expert. Both are free softwares and are easy to install/configure. Zabbix does have a cuter interface though – can become handy if your client require access.
Both software allows sysadmins to run remote command. Personally, I find those systems to be way too complicated to setup when Monit is easily available. Its configuration allow a syntax very similar to : if load > 5 for 10 minutes, then stop postfix-delivery. Another life saver when you don’t expect your remote monitoring agent to be able to launch a command. I use it for limits like (if load>80 for 2 minutes, then stop {httpd,mysqld}). If your system is badly losing interactivity, your normal remote monitoring software will never be able to save your system (ssh will timeout).
3. Log/Security software
While Zabbix/nagios can do some checksum on important files (such as /etc/passwd, /etc/shadow, …), they are not ready as IDS (Intrusion Detection System) yet. For such system, I recommend OSSEC. Following the online documentation, you will have a log-analysis system created in no time – using thousand of rules given with the software. Customization can also be done pretty quickly. The ‘action’ following a trigger can be email-alert or a command. The system come with a pre-built interface to iptables… port-scanning and brute force password testing are no more.
Add to all these tools a svn repository for your code, an Puppet system for global configuration and some wiki for documentation and you should have a pretty strong backbone to deal with anything your clients throws at you.
removing mysql-bin log files
One of the main part of Laboratoires Phoenix sentinel network is the Zabbix monitoring system.
In direct correlation with this fact is that the main concern inside this sentinel network is the database footprint of MySQL. I do not mean the ‘size-in-memory’, since I do have quite enough ram on those systems. And I’m not doing that much caching since data change very, very often. What I mean is the size of the database & binaries log-files on disk.
To make the story short: I always move /var/lib/mysql on a separate partition to be certain a db surge would not bring down / compromise other server functions. Seem like I forgot (on one of the system) that Debian standard location for the mysql-bin log-files was in /var/log/mysql.
Since those are independent servers (not multi-master / replicated MySQL system), here is the magic sequence to remove unused mysql-bin files.
# vi /etc/mysql/my.cnf
[modify bin-log number/space usage]
# ls -la /var/log/mysq/
# mysql -uroot -p
[enter password]
# purge binary logs to ‘mysql-bin.000321′;
[where mysql-bin.000321 is one of the last / up to the point you want too keep].
There you go.