“People who know a little bit of statistics – enough to use statistical techniques, not enough to understand why or how they work – often end up horribly misusing them.  Statistical tests are complicated mathematical techniques, and to work, they tend to make numerous assumptions. The problem is that if those assumptions are not valid, most statistical tests do not cleanly fail and produce obviously false results.”

“People who know a little bit of IT Security – enough to use an IDS or SIEM, not enough to understand why or how they work – often end up horribly misusing them.  Security tools use complicated technical techniques, and to work, they tend to make numerous assumptions. The problem is that if those assumptions are not valid, most security tools do not cleanly fail and produce obviously false results.”

Syslog-ng facilitates easy integration with Perl binaries as the Perl program is spawned once during the daemon start up and a handle to that program’s STDIN is maintained for dispatching of messages. Using POE, we can turn this into an event driven model, making additional complexity simple.

In this example, we’ll create a POE Master session that receives all of the syslog-ng input from STDIN. Using off the shelf components, we’ll run a TCP Server on port 9514 that will allow clients to connect and subscribe to feeds based on the “program” name of the message being dispatched.

Anytime I’m using Regular Expressions over and over, I like to “precook” them. This compiles the regular expression, and lets the engine skip that step each time they’re used. Doing so is simply a matter of declaring the regex with the qr// operator:

my %cooked = (
	program => qr/\s+\d+:\d+:\d+\s+\S+\s+([^:\s]+)(:|\s)/,
);

Initialization

Next we’ll create the administrative session in charge of dispatching the messages to the proper channels:

# Dispatcher Master Session
POE::Session->create(
	inline_states => {
		_start					=> \&dispatcher_start,
		_stop					=> sub { print "SESSION ", $_[SESSION]->ID, " stopped.\n"; },
		register_client			=> \&register_client,
		subscribe_client		=> \&subscribe_client,
		hangup_client			=> \&hangup_client,

		dispatch_message		=> \&dispatch_message,
	},
);

We’ll define those subroutines shortly, but we need to setup the rest of our sessions. Next, we’ll need a TCP Server to handle the client connections, we can get that using POE::Component::Server::TCP:

# TCP Session Master
POE::Component::Server::TCP->new(
		Alias		=> 'server',
		Address		=> '127.0.0.1',
		Port		=> 9514,

		ClientConnected		=> \&client_connect,
		ClientInput			=> \&client_input,

		ClientDisconnected	=> \&client_term,
		ClientError			=> \&client_term,

		InlineStates		=> {
			client_print		=> \&client_print,
		},
);

The final session will handle the Input on STDIN from syslog-ng:

# Syslog-ng Stream Master
POE::Session->create(
		inline_states => {
			_start		=> \&stream_start,
			_stop		=> sub { print "SESSION ", $_[SESSION]->ID, " stopped.\n"; },
			stream_line		=> \&stream_line,
		},
);

Now we have to define the subroutines that we’ll be dispatching events to. The heavy lifting is done by POE, and we’re left to handle simple things.

Session Routines: dispatcher

This session is going to managing which clients receive which messages. The actual input is handled by the stream session, and the sending of the messages to the client by the server session. As we have a raw POE::Session, our first subroutine dispatcher_start is just going to do some basic preparation:

sub dispatcher_start {
	my ($kernel, $heap) = @_[KERNEL, HEAP];

	$kernel->alias_set( 'dispatcher' );  # allow named dispatch to this session.

	$heap->{subscribers} = {};
        $heap->{clients} = {};

}

Next event to be handled is the register_client event which is fired anytime a connection is established to the server session. All the dispatcher does is register it’s session_id into an internal heap. Nothing happens with it, but if we needed to send a message to all clients, we could loop over this hash and broadcast message.

sub register_client {
    # ARG0 => TCP Client Session ID
    my ($kernel,$heap,$sid) = @_[KERNEL,HEAP,ARG0];

    $heap->{clients}{$sid} = 1;
}

Clients can subscribe to a program name, which they do by entering “sub dhcpd, dnsmasq” into the TCP Server. It’s not fancy, but man is it convenient for debugging and development purposes. The server session determines that the subscription is occurring and passes it’s argument string to the dispatcher session via the subscribe_client event. This subroutine is called:

sub subscribe_client {
    # ARG0 => SID of Client
    # ARG1 => Argument String of the subscribe
	my ($kernel,$heap,$sid,$argstr) = @_[KERNEL,HEAP,ARG0,ARG1];

    # Split the input at commas or spaces into words:
	my @progs = map { lc } split /[\s,]+/, $argstr;
    # Add the SID to the list of Subscribed Clients for that program
	foreach my $prog (@progs) {
		$heap->{subscribers}{$prog}{$sid} = 1;
	}

    # Inform the client they've subscribed via client_print
	$kernel->post( $sid => 'client_print' => 'Subscribed to : ' . join(', ', @progs ) );
}

If a client disconnects, we remove it from the message dispatching hash:

sub hangup_client {
    # ARG0 => SID of Client Disconnecting
	my ($kernel,$heap,$sid) = @_[KERNEL,HEAP,ARG0];

	delete $heap->{clients}{$sid};

	foreach my $p ( keys %{ $heap->{subscribers} } ) {
		delete $heap->{subscribers}{$p}{$sid}
			if exists $heap->{subscribers}{$p}{$sid};
	}
}

Now comes the most important event the dispatcher handles, dispatch_message. In this event, we have a message from syslog-ng that needs to go to interested parties. This event determines the “program” and it’s subscribers and sends that message along appropriately:

sub dispatch_message {
    # ARG0 => The raw message from syslog-ng
	my ($kernel,$heap,$msg) = @_[KERNEL,HEAP,ARG0];

    # Determine the program name
	if( my ($program) = map { lc } ($msg =~ /$cooked{program}/) ) {
		# remove the sub process and PID from the program
		$program =~ s/\(.*//g;
		$program =~ s/\[.*//g;

        # If we have subscribers, send them the message.
		if( exists $heap->{subscribers}{$program} ) {
			foreach my $sid (keys %{ $heap->{subscribers}{$program} }) {
				$kernel->post( $sid => 'client_print' => $msg );
			}
		}
}

You'll notice on line 14 above, the post( $sid => client_print => $msg ) sends the event to the appropriate client and calls the client_print event on itself. This is all the dispatcher needs to do. The rest is handled by other other sessions.

Session Routines: server

This session accepts new tcp clients and handles writing to the sockets. We'll take a look at a few subroutines here. Fist we'll look at the ClientConnect event.

sub client_connect {
    # SESSION is the client's session object
	my ($kernel,$heap,$ses) = @_[KERNEL,HEAP,SESSION];

	my $SID = $ses->ID;

    # Register the Client with the Dispatcher
	$kernel->post( 'dispatcher' => 'register_client' => $SID );

    # Store the current entry for 'client' in the heap so we can communicate later
	$heap->{clients}{ $SID } = $heap->{client};

	# Say hello to the client.
	$heap->{client}->put( "Hello Client: $SID" );
}

We also need a disconnect event:

sub client_term {
	my ($kernel,$heap,$ses) = @_[KERNEL,HEAP,SESSION];
	my $sid = $ses->ID;

    # Delete the Client's Dispatch Table
	delete $heap->{dispatch}{$sid};
    # Tell the dispatcher session we're through
	$kernel->post( 'dispatcher' => 'hangup_client' =>  $sid );
}

Next we’ll handle sending message to the client, which is incredibly easy:

sub client_print {
    # ARG0 => Message to Send to the Client
	my ($kernel,$heap,$ses,$mesg) = @_[KERNEL,HEAP,SESSION,ARG0];

	$heap->{clients}{$ses->ID}->put($mesg);
}

Now we a routine to handle the ClientInput event. This event will take commands from the clients and do something with them. We’ll use an internal dispatch table in the form of a hash to handle translating commands. This will allow us to expand our API if we need to.

sub client_input {
    # SESSION is the Client Session Object with input
    # ARG0 => Input waiting from that client
	my ($kernel,$heap,$ses,$msg) = @_[KERNEL,HEAP,SESSION,ARG0];
	my $sid = $ses->ID;

    # Build a Dispatch Table if one does not exists in the heap for this entry.
	if( !exists $heap->{dispatch}{$sid} ) {
		$heap->{dispatch}{$sid} = {

			subscribe		=> {
				re			=> qr/^sub(?:scribe)? (.*)/,
				callback	=> sub {
					$kernel->post( 'dispatcher' => 'subscribe_client' => $sid, shift );
				},
			},
            # FUTURE API for Clients receiving every message!
			#fullfeed		=> {
			#	re			=> qr/^(fullfeed)/,
			#	callback	=> sub {
			#		$kernel->post( 'dispatcher' => 'fullfeed_client' => $sid );
			#	},
			#},
		};
	}

	#
	# Check for messages:
	my $handled = 0;
    # Get Our Dispatch Table
	my $dispatch = $heap->{dispatch}{$sid};
    # Look up and take action according to our dispatch table
	foreach my $evt ( keys %{ $dispatch } ) {
		if( my($args) = ($msg =~ /$dispatch->{$evt}{re}/)) {
			$handled = 1;
			$dispatch->{$evt}{callback}->($args);
			last;
		}
	}

    # Inform the client that their command was not understood.
	if( !$handled ) {
		$kernel->post( $sid => 'client_print' => 'UNKNOWN COMMAND, Ignored.' );
	}
}

That’s the most complicated routine in the program, but it does allow us to morph the dispatch tables for individual clients. Lines 12-15 build a dispatch table entry with the regular expression to match the command, followed by a callback subroutine reference which handles the command. Lines 34 and 36 are where these rules are applied to the input from the client.

Session Routines: stream

The last session is very simple. This session maintains the connection to STDIN from syslog-ng and dispatches those lines as events to the dispatcher session. There is a startup routine:

sub stream_start {
	my ($kernel, $heap) = @_[KERNEL, HEAP];

	$kernel->alias_set( 'stream' );

	#
	# Initialize the connection to STDIN as a POE::Wheel
	my $stdin = IO::Handle->new_from_fd( \*STDIN, 'r' );
	my $stderr = IO::Handle->new_from_fd( \*STDERR, 'w' );

	$heap->{stream} = POE::Wheel::ReadWrite->new(
		InputHandle		=> $stdin,
		OutputHandle	=> $stderr,
		InputEvent		=> 'stream_line',
	);
}

And the stream_line event which sends the incoming syslog messages to the dispatcher session for processing:

#--------------------------------------------------------------------------#
sub stream_line {
    # ARG0 => Line from STDIN, New line delimited.
	my ($kernel,$msg) = @_[KERNEL,ARG0];

	return unless length $msg;

	$kernel->post( 'dispatcher' => 'dispatch_message' => $msg );

}

Setting it up with syslog-ng

If we store our POE program in /usr/local/bin/poe-syslog-ng.pl, in the syslog-ng.conf we need to specify it as a program:

#
# Subscriber Feeds
destination d_subscribers {
	program("/usr/local/bin/poe-syslog-ng.pl");
};

Then you can feed it based on filters, just like the rest of the destination macros in syslog-ng:

#
# SUBSCRIPTION SERVICE:
log { source(s_ext); source(s_udp); filter(f_database); destination(d_subscribers); };

The whole #!

For those interested, I’ve written a program that expands this example with enhanced functionality. The full source is available here:

#!/usr/bin/perl
#
# This is the POE Master Server.
#  1) Take all the syslog input
#  2) Listen for parsers
#  3) Filter streams to parsers
#  TODO: 4) Maintain Parser State, restarting on crash

use strict;
use warnings;

use Socket;
use Regexp::Common qw(net);

sub POE::Kernel::ASSERT_DEFAULT (){ 1 }
#sub POE::Kernel::TRACE_DEFAULT (){ 1 }
use POE qw(
	Wheel::ReadWrite
	Component::Server::TCP
);

my %cooked = (
	program => qr/\s+\d+:\d+:\d+\s+\S+\s+([^:\s]+)(:|\s)/,
);

#--------------------------------------------------------------------------#
# POE Session Initialization

# Dispatcher Master Session
POE::Session->create(
	inline_states => {
		_start					=> \&dispatcher_start,
		_stop					=> sub { print "SESSION ", $_[SESSION]->ID, " stopped.\n"; },
		register_client			=> \&register_client,
		subscribe_client		=> \&subscribe_client,
		unsubscribe_client		=> \&unsubscribe_client,
		fullfeed_client			=> \&fullfeed_client,
		dispatch_message		=> \&dispatch_message,
		broadcast				=> \&broadcast,
		hangup_client			=> \&hangup_client,
		server_shutdown			=> \&server_shutdown,
		debug_client			=> \&debug_client,
		nobug_client			=> \&nobug_client,
		debug_message			=> \&debug_message,
	},
);

# TCP Session Master
POE::Component::Server::TCP->new(
		Alias		=> 'server',
		Address		=> '127.0.0.1',
		Port		=> 9514,

		ClientConnected		=> \&client_connect,
		ClientInput			=> \&client_input,

		ClientDisconnected	=> \&client_term,
		ClientError			=> \&client_term,

		InlineStates		=> {
			client_print		=> \&client_print,
		},
);

# Syslog-ng Stream Master
POE::Session->create(
		inline_states => {
			_start		=> \&stream_start,
			_stop		=> sub { print "SESSION ", $_[SESSION]->ID, " stopped.\n"; },

			stream_line		=> \&stream_line,
			stream_error	=> \&stream_error,
		},
);

#--------------------------------------------------------------------------#

#--------------------------------------------------------------------------#
# POE Main Loop
POE::Kernel->run();
exit 0;
#--------------------------------------------------------------------------#

#--------------------------------------------------------------------------#
# POE Event Functions
#--------------------------------------------------------------------------#

#--------------------------------------------------------------------------#
sub debug {
	my $msg = shift;
	chomp($msg);
	$poe_kernel->post( 'dispatcher' => 'debug_message' => $msg );
	print "[debug] $msg\n";
}
#--------------------------------------------------------------------------#
sub dispatcher_start {
	my ($kernel, $heap) = @_[KERNEL, HEAP];

	$kernel->alias_set( 'dispatcher' );

	$heap->{subscribers} = { };
	$heap->{full} = { };
	$heap->{debug} = { };
}

#--------------------------------------------------------------------------#
sub register_client {
	my ($kernel,$heap,$sid) = @_[KERNEL,HEAP,ARG0];

	$heap->{clients}{$sid} = 1;
}

#--------------------------------------------------------------------------#
sub debug_client {
	my ($kernel,$heap,$sid) = @_[KERNEL,HEAP,ARG0];

	if( exists $heap->{full}{$sid} ) {  return;  }

	$heap->{debug}{$sid} = 1;
	$kernel->post( $sid => 'client_print' => 'Debugging enabled.' );
}

#--------------------------------------------------------------------------#
sub nobug_client {
	my ($kernel,$heap,$sid) = @_[KERNEL,HEAP,ARG0];

	delete $heap->{debug}{$sid}
		if exists $heap->{debug}{$sid};
	$kernel->post( $sid => 'client_print' => 'Debugging disabled.' );
}

#--------------------------------------------------------------------------#
sub fullfeed_client {
	my ($kernel,$heap,$sid) = @_[KERNEL,HEAP,ARG0];

	#
	# Remove from normal subscribers.
	foreach my $prog (keys %{ $heap->{subscribers} }) {
		delete $heap->{subscribers}{$prog}{$sid}
			if exists $heap->{subscribers}{$prog}{$sid};
	}

	#
	# Turn off DEBUG
	if( exists $heap->{debug}{$sid} ) {
		delete $heap->{debug}{$sid};
	}

	#
	# Add to fullfeed:
	$heap->{full}{$sid} = 1;

	$kernel->post( $sid => 'client_print' => 'Full feed enabled, all other functions disabled.');
}

#--------------------------------------------------------------------------#
sub subscribe_client {
	my ($kernel,$heap,$sid,$argstr) = @_[KERNEL,HEAP,ARG0,ARG1];

	if( exists $heap->{full}{$sid} ) {  return;  }

	my @progs = map { lc } split /[\s,]+/, $argstr;
	foreach my $prog (@progs) {
		$heap->{subscribers}{$prog}{$sid} = 1;
	}

	$kernel->post( $sid => 'client_print' => 'Subscribed to : ' . join(', ', @progs ) );
}
#--------------------------------------------------------------------------#
sub unsubscribe_client {
	my ($kernel,$heap,$sid,$argstr) = @_[KERNEL,HEAP,ARG0,ARG1];

	my @progs = map { lc } split /[\s,]+/, $argstr;
	foreach my $prog (@progs) {
		delete $heap->{subscribers}{$prog}{$sid};
	}

	$kernel->post( $sid => 'client_print' => 'Subscription removed for : ' . join(', ', @progs ) );
}

#--------------------------------------------------------------------------#
sub hangup_client {
	my ($kernel,$heap,$sid) = @_[KERNEL,HEAP,ARG0];

	delete $heap->{clients}{$sid};

	foreach my $p ( keys %{ $heap->{subscribers} } ) {
		delete $heap->{subscribers}{$p}{$sid}
			if exists $heap->{subscribers}{$p}{$sid};
	}

	if( exists $heap->{debug}{$sid} ) {
		delete $heap->{debug}{$sid};
	}

	if( exists $heap->{full}{$sid} ) {
		delete $heap->{full}{$sid};
	}

	debug("Client Termination Posted: $sid\n");

}

#--------------------------------------------------------------------------#
sub stream_start {
	my ($kernel, $heap) = @_[KERNEL, HEAP];

	$kernel->alias_set( 'stream' );

	#
	# Initialize the connection to STDIN as a POE::Wheel
	my $stdin = IO::Handle->new_from_fd( \*STDIN, 'r' );
	my $stderr = IO::Handle->new_from_fd( \*STDERR, 'w' );

	$heap->{stream} = POE::Wheel::ReadWrite->new(
		InputHandle		=> $stdin,
		OutputHandle	=> $stderr,
		InputEvent		=> 'stream_line',
		ErrorEvent		=> 'stream_error',
	);
}

#--------------------------------------------------------------------------#
sub stream_line {
	my ($kernel,$msg) = @_[KERNEL,ARG0];

	return unless length $msg;

	$kernel->post( 'dispatcher' => 'dispatch_message' => $msg );

}

#--------------------------------------------------------------------------#
sub stream_error {
	my ($kernel) = $_[KERNEL];

	debug("STREAM ERROR!!!!!!!!!!\n");
	$kernel->call( 'dispatcher' => 'server_shutdown' => 'Stream lost' );
}

#--------------------------------------------------------------------------#
sub server_shutdown {
	my ($kernel,$heap,$msg) = @_[KERNEL,HEAP,ARG0];

	$kernel->call( dispatcher => 'broadcast' => 'SERVER DISCONNECTING: ' . $msg );
	$kernel->call( 'server' => 'shutdown' );
	exit;
}

#--------------------------------------------------------------------------#
sub client_connect {
	my ($kernel,$heap,$ses) = @_[KERNEL,HEAP,SESSION];

	my $KID = $kernel->ID();
	my $CID = $heap->{client}->ID;
	my $SID = $ses->ID;

	$kernel->post( 'dispatcher' => 'register_client' => $SID );

	$heap->{clients}{ $SID } = $heap->{client};
	#
	# Say hello to the client.
	$heap->{client}->put( "EHLO Streamer (KERNEL: $KID:$SID)" );
}

#--------------------------------------------------------------------------#
sub client_print {
	my ($kernel,$heap,$ses,$mesg) = @_[KERNEL,HEAP,SESSION,ARG0];

	$heap->{clients}{$ses->ID}->put($mesg);
}

#--------------------------------------------------------------------------#
sub broadcast {
	my ($kernel,$heap,$msg) = @_[KERNEL,HEAP,ARG0];

	foreach my $sid (keys %{ $heap->{clients} }) {
		$kernel->post( $sid => 'client_print' => $msg );
	}
}
#--------------------------------------------------------------------------#
sub dispatch_message {
	my ($kernel,$heap,$msg) = @_[KERNEL,HEAP,ARG0];

	foreach my $sid ( keys %{ $heap->{full} } ) {
		$kernel->post( $sid => 'client_print' => $msg );
	}

	if( my ($program) = map { lc } ($msg =~ /$cooked{program}/) ) {
		# remove the sub process and PID from the program
		$program =~ s/\(.*//g;
		$program =~ s/\[.*//g;

		debug("DISPATCHING MESSAGE [$program]");

		if( exists $heap->{subscribers}{$program} ) {
			foreach my $sid (keys %{ $heap->{subscribers}{$program} }) {
				$kernel->post( $sid => 'client_print' => $msg );
			}
		}
		else {
			debug("Message discarded, no listeners.");
		}
	}
	else {
			debug("Message discarded, format not understood.");
	}
}

#--------------------------------------------------------------------------#
sub debug_message {
	my ($kernel,$heap,$msg) = @_[KERNEL,HEAP,ARG0];

	foreach my $sid (keys %{ $heap->{debug} }) {
		$kernel->post( $sid => 'client_print' => '[debug] ' . $msg );
	}
}

#--------------------------------------------------------------------------#
sub client_input {
	my ($kernel,$heap,$ses,$msg) = @_[KERNEL,HEAP,SESSION,ARG0];
	my $sid = $ses->ID;

	if( !exists $heap->{dispatch}{$sid} ) {
		$heap->{dispatch}{$sid} = {
			fullfeed		=> {
				re			=> qr/^(fullfeed)/,
				callback	=> sub {
					$kernel->post( 'dispatcher' => 'fullfeed_client' => $sid );
				},
			},
			subscribe		=> {
				re			=> qr/^sub(?:scribe)? (.*)/,
				callback	=> sub {
					$kernel->post( 'dispatcher' => 'subscribe_client' => $sid, shift );
				},
			},
			unsubscribe 	=> {
				re			=> qr/^unsub(?:scribe)? (.*)/,
				callback	=> sub {
					$kernel->post( 'dispatcher' => 'unsubscribe_client' => $sid, shift );
				},
			},
			debug 	=> {
				re			=> qr/^(debug)/i,
				callback	=> sub {
					$kernel->post( 'dispatcher' => 'debug_client' => $sid, shift );
				},
			},
			nobug 	=> {
				re			=> qr/^(no(de)?bug)/i,
				callback	=> sub {
					$kernel->post( 'dispatcher' => 'nobug_client' => $sid, shift );
				},
			},
			#quit			=> {
			#	re			=> qr/(exit)|q(uit)?/,
			#	callback	=> sub {
			#			$kernel->post( $sid => 'client_print' => 'Terminating connection on your request.');
			#			$kernel->post( $sid => 'shutdown' );
			#	},
			#},
			#status			=> {
			#	re			=> qr/^status/,
			#	callback	=> sub {
			#		my $cnt = scalar( keys %{ $heap->{clients} } );
			#		my $subcnt = scalar( keys %{ $heap->{subscribers} });
			#		my $msg = "Currently $cnt connections, $subcnt subscribed.";
			#		$kernel->post( $sid, 'client_print', $msg );
			#	},
			#},
		};
	}

	#
	# Check for messages:
	my $handled = 0;
	my $dispatch = $heap->{dispatch}{$sid};
	foreach my $evt ( keys %{ $dispatch } ) {
		if( my($args) = ($msg =~ /$dispatch->{$evt}{re}/)) {
			$handled = 1;
			$dispatch->{$evt}{callback}->($args);
			last;
		}
	}

	if( !$handled ) {
		$kernel->post( $sid => 'client_print' => 'UNKNOWN COMMAND, Ignored.' );
	}
}

#--------------------------------------------------------------------------#
sub client_term {
	my ($kernel,$heap,$ses) = @_[KERNEL,HEAP,SESSION];
	my $sid = $ses->ID;

	delete $heap->{dispatch}{$sid};
	$kernel->post( 'dispatcher' => 'hangup_client' =>  $sid );

	debug("SERVER, client $sid disconnected.\n");
}

#--------------------------------------------------------------------------#

I’ve been working on packaging the code that I have into something that might be useful to the general public. I’ll post another blog entry when I have a rough cut version of the package available for testing/breaking.

Then this weekend, the data center that hosts this server exploded. The site is back up and running now, but there should be a few more hours of downtime on the horizon as they install and integrate a permanent electrical infrastructure to the data center.

Also, I’ve been selected to speak at the Linux World Expo in San Fransisco this year! My talk is “Network Introspection with Open Source Tools.” If you’re going, please stop by and heckle me!

I may start updating this blog at some point.

OpenSSH is a glorious implementation of a critical network protocol. Most networks have disabled and banned the use of telnet, rsh, and ftp in favor of the more “secure” SSH protocol. OpenSSH runs on every platform I’ve encountered (using CygWin on Windows). SSH provides an encrypted channel for data transfer. Usually that’s Keyboard Interactive Sessions or Files (using SCP), however SSH is capable of setting up multiple channels and acting as a SOCKS4 or SOCKS5 Proxy.

From the manual:

-D [bind_address:]port

Specifies a local ``dynamic'' application-level port forwarding.
This works by allocating a socket to listen to port on the local
side, optionally bound to the specified bind_address.  Whenever a
connection is made to this port, the connection is forwarded over
the secure channel, and the application protocol is then used to
determine where to connect to from the remote machine.  Currently
the SOCKS4 and SOCKS5 protocols are supported, and ssh will act
as a SOCKS server.  Only root can forward privileged ports.  Dy-
namic port forwardings can also be specified in the configuration
file.

That’s a lot of technical mumbo jumbo, so what does it mean?! Well it means that if you have ssh and an ssh server outside of your work network that you can connect to, you can SOCKS5 Proxy all your Interesting Traffic elsewhere by issuing this command:

ssh -D3128 server

Then pointing your applications to SOCKS5 Proxy localhost port 3128 will encrypt all the traffic between you and the server that you’re connecting to. This only provides privacy from the LOCAL or CORPORATE network, and does not encrypt your traffic on it’s way to it’s external destinations!!!!

In the next installment, we’ll cover PuTTY.exe and how to evade proxies on Windows platforms.

I’ll be continuing my Proxy Evasion series as soon as I get a chance to put together some screen shots for the tutorial part of the article.

Hopefully that article will be completed relatively soon.

Hope everyone had a great holiday season.

Pick up a copy of this book:

Additionally, these proxies do very little to increase the security of the network. I’m fairly certain that by the time the Proxy Vendor is alerted to malware distributing websites, an anti-virus company has already issued an update. This is redundant. Normally, I’d fully support this redundant – ok, I’ll drop the buzzword – Defense in Depth solution.

However, piggy-backing on the heels of ay real security value, which is best described as the graph of 1/x, are made up categories of websites to deter your users from doing things that your CEO believes are inappropriate. Interestingly enough, if the CEO is involved in Fantasy Football, you’ll be hard pressed to find an IT Infrastructure that denotes that classification of sites as inappropriate. I digress.

I hate these policies. The whole concept of the internet is free access to information. As my job is Information Security, I frequently surf into the areas of the internet that WebSense might classify as “Inappropiate Content”, “Hacker Sites”, or “Proxy Sites”. It’s the nature of my business. Luckily for me, I’ve been granted an exception to the policy that allows me to view such terrible web content.

However, my users are frequently inconvenienced by searches for “adult oriented material” as some of our reproductivity scientists might need access to sites that contain terms like “sex”, with interesting prefixes like “oral” and “vaginal”. *Gasp*

So, tired of executives so out of touch with their users that they don’t recognize them, much less know what they do, I begin my multipart series on Proxy Evasion with the Environmental Concerns.

CygWin

If you’re running Windows, I highly recommend that you install CygWin. It provides a POSIX Compliant Environment for Windows. I’d be lost on Windows without it.

Using CygWin you can install a host of tools for network scouting, monitoring, manipulation, defense, and attack. Some of my indispensables include:

• nmap – find out about a host
• iptraf – find out about network traffic
• tcpdump / libpcap – excellent network sniffer
• winpcap – I can’t remember if CygWin actually has libpcap support, if not, I remember having enormous success with WinPCAP

None of those are necessary for Proxy Evasion, but they are nice tools to have laying around when you’re connected to a network.

Mac OS X

Don’t worry, I’m a Mac user too. Mac OS X comes with a number of UNIX utilities already installed. I highly reccommend installing the Developer’s Tools package to get GCC and then installing one of the ports systems available. The two forerunners in the GNU/OpenSource porting for OS X are:

• Fink – Provides source & binary downloads of packages with full dependency support.
• MacPorts – Previously DarwinPorts, source only ports system.

Though most free software will download and compile without hassle, it’s nice to have a package management suite that manages and downloads dependencies so you spend less time searching and installing and more time using your software.

But.. But.. I don’t have Admin Rights!

As a way to “increase security”, organizations will remove administrative privileges from average user’s computers. I feel this is complete hipocracy, so I discourage it. However, if you’re unfortunate to have these restraints enforced on your computer, there are ways to run your programs without installing them, and thus be compliant with the “I will not install my own software on company computers” rule you signed when you took the job.

They’re called “Portable Apps.” They’re designed to be installed & run off of USB flash drives and require no disk access on the computer you run it on. This generally avoids all automated software policy enforcement, allowing you to run your programs without being hassled.

Here are the two biggest repositories for portable apps:

• Portable Windows Apps
• Portable Mac OS X Apps

This biggest advantage to portable apps is the fact that they’re preferences are also stored on the drives. This means even if you don’t have access to modify the network settings (ie, Proxy settings) on your applications because of an enforced policy, you can still modify the preferences on the portable apps. This is terribly useful once we have tunnels setup to use for proxying.

I carry around a copy of Thunderbird, Firefox, Gaim/Adium, Abiword, and for Windows, PuTTY.

If you’re on Windows, please download PuTTY now. It’s a light weight ssh terminal that does not need to be installed. It’s precompiled and can run with out writing preferences anywhere you’re not allowed.

By becoming aware of software solutions that allow you to do your work, you can setup a hospitable environment for productivity, free from the annoyances of “ADMINISTRATOR PRIVILEGES REQUIRED.”

In the next article, we’ll cover using SSH for Proxy Evasion.

There’s been a lot of coverage for a while about the inconsistencies and problems with the U.S. and it’s failed attempts at providing real security. We have Airport Screeners missing 20 of 22 bombs in a live test of security conducted by TSA. The New Scientist argues that most of the $44 BILLION spent on bioterrorism defense has been wasted. Recently, a bright young student at University of Indiana made a fake boarding pass program. The interesting thing is this sort of thing had been documented several times before.

Blame, of course, fell on this unsuspecting young student. He crossed the line. He made it _EASY_ for people to do this. See, that’s the problem with these terrorists. They’re inherently lazy. They have no drive or ambition that would push them to open up MS Paint and digitally alter the image of a boarding pass. That would require about 15 minutes, and terrorists certainly don’t have 15 minutes to spare to circumvent poor security measures. The blame is misplaced. TSA should be raided by the FBI. Their lives should be disrupted and their houses sacked.

But we won’t point the finger at our security measures, certainly not Airport Security. The fact is that good security measures could be developed that introduce little inconvenience into the picture for the end users. However, I’m not entirely sure that would make people feel better. This isn’t about real security or fighting terrorism. It’s about being affected on a personal level by these security measures.

I don’t fly much. Not because I’m afraid of terrorists, I rarely have the need to fly. However, when I do, and I’m stuck in the long security lines at Airports, I can’t help but over hear people talking about how “at least we know we’re safe on the plane.” Depending on my relative proximity to these conversations, I will tend to pipe up and explain the fault with the systems to my fellow travellers. The response has not been good. Now, they’re standing in a long line, wasting time, and Airport Screeners missed 20 out of 22 bombs. Oddly enough, they kill the messenger. Most of them just ask me to stop talking and then mutter under their breaths about how I don’t really know what I’m talking about.

They want to feel secure, not be secure. For these people, and they represent the masses, Security Theatre is not just good enough, it’s the requirement. They may complain about it and rant, but if after 9/11 (take a shot if you’re playing along at home), they weren’t forced to stand in ridiculous lines and have underpaid, uninterested security guards look at their personal effects and throw their lip gloss in a trashcan labeled ‘contraband’, they’d be disoriented and petrified. Sure, you think they’re scared now, but I really don’t think that air travel would’ve recovered as quickly as it did post 9/11 (*shot*) if people hadn’t been grotesquely inconvenienced.

There’s a problem with the perception of Security. I see it in my day to day duties as a System Administrator / Programmer / Security Administrator. If I had a nickel for every time I heard “security is getting in my way,” I’d be set. People perceive security as inconvenience these days. If they’re not being inconvenienced, then they’re pretty sure they’re not secure. Nothing is going to change with the large scale security systems in this country until we change the perception of security.

There are two interpretations of this memo.

• Encrypt the ENTIRE disk.
• Encrypt just the files containing the data.

So, what’s a lowly security administrator to do?! Choices are bad! Obviously you encrypt the entire disk! Right?! no? Why not?

Well, recently concerned has been growing in the media over “Personally Identifiable Information” being recovered from stolen laptops. Sadly, they’ve missed the point. You see, most of the identity theft perpitrated results from malware, with a smaller number coming from physical security breaches, involving mostly server hardware like backup tapes, hard drives, and entire computers. OMB and the Presidential mandate don’t deal with any of these issues, so their mandates can be viewed as little more than knee-jerk reactions to news coverage on the stolen VA Laptop.

Essentially, the media is now responsible for dictating Federal IT Security Policy. This is what happens when you have governing bodies like OMB that don’t rationally evaluate threats or understand the technical aspect of IT Security. It takes maybe another 10 minutes of searching through the archives at Emergent Chaos to realize that most breaches are the result of software breakins. However, that’s not gonna stop the Federal Government from shelling out millions, possibly billions, to address the threat of data being stolen from a laptop that’s shutdown.

I don’t know about you, but the last time my laptop was shutdown was, well.. that one time it ran out of battery and I was miles away from a power adapter. Otherwise my PowerBook just gets folded up and goes to sleep. Full Disk Encryption decrypts the disk at BOOT. So, since I’ve already booted, my entire drive is already booted, I gain nothing unless the battery dies.

“Full Disk Encryption” is also a pretty intimidating mouthful for most computer users. Uninitiated, and some who think they’re initiated, sporting CISSP’s, would be lulled into thinking “HEY! My WHOLE disk is encrypted! I’m secure!!!!!!!!!” Unfortunately, this does nothing to protect your data from the software threats that are much more common. You see, the disk is decrypted at boot, and then any programs just use the disk without even knowing that it’s encrypted. So all the viruses and malware you’ve accumulated surfing the net for discount shopping and myspace.com updates on IE, is able to read ALL the data on your drive.

You haven’t really secured things from the most common threat, however, you have added another layer of complexity to the user’s experience.

So what is the solution? Well first, it’s time to start investigating new methods for virus detection. The Big 3 Vendors (Symantec, McAfee, and Trend Micro) have miss rates of 80% because Virus authors are testing their virus against them. Closing this hole in the organizational structure will eliminate 80% of the threat to Identity Theft.

Horrible programming practices are usually to blame for the majority of personalized attacks that have leaked information in the past. Managers should be encouraged to hire talented programmers and work with the programmers to create an atmosphere of cooperation. The programmers should be involved in the design process. They should also be given the right to veto or question their managers decisions. Source control systems should be in place and encouraged. Peer reviews should be factored into the development process. The more eyes on the code, the more likely something will be caught. The organization should adapt Best Practices based on recommendations by the development team. These best practices require the same peer review that the code base gets.

This sounds like a lot of work, and it is. Additionally, it’ll only fix like 1% of the Identity Theft problems. However, it will raise the quality of the code, thus the product. It might initially introduce some overhead, but that overhead will pay for itself and prove more cost effective as the products developed more accurately reflect customer desires.

If you have people with sensitive data on laptops or other portable media, you’re gonna need to deploy some form of encrypted mechanism for storage. Personally, the encryption schemes that come builtin to Mac OS X and Windows XP should suffice for most intents and purposes. Even OMB could’ve saved some money by leveraging this had they paid attention to their own rules. Apple and Microsoft are both in process for attaining the coveted FIPS-140-2 compliance that is required for productions imploring encryption of federal data. The Apple and Microsoft solutions have no cost overhead as they’re already installed on all the Apple and Microsoft laptops in production.

Folders with sensitive data should be encrypted in such a manner that there’s a an inactivity timeout, and the files must be decrypted when required. Apple’s Disk Utility allows a user to construct an encrypted disk image that can be mounted like a regular DMG. I’ve been told that Windows XP has a similar utility. There are also free products out there like TrueCrypt that allow you to hide the encrypted image in a JPG or other benign file. For most people, the built-in encryption tools should be sufficient.

Users need to be trained to use the built-in features. That’s where the money could go. A simple PowerPoint presentation would satisfy most users. I’d recommend that people routinely working with sensitive data be instructed on proper ways to store that data on their local machines. You can pretend it won’t happen by making it a violation of Policy, but policy is a horrible place to hinge your IT Security Infrastructure on.

Don’t be sucked into the hype. Think about things rationally and don’t make mandates that affect all government organizations without figuring out if there’s a potential threat there. Realize, in my organization, we have over 300 users with laptops and in two years, we’ve had 0 lost or stolen laptops.

ZeFrank on Terrorism

It’s really that simple. Stop being terrorized. Stop being scared to live. Stop taking life so seriously, you’re never gonna make it out alive. We don’t need billions of dollars of security screening software/hardware. Anyone with a week of spare time will be able to circumvent it anyways. This security is just a show, and I’m not entertained in the slightest.

I’m not flying again until these ridiculous regulations stop. We know we’re accepting a risk getting onto a plane. We’re 30,000 feet in the air, and if something mechanical fails, that’s a LONG way down. You’re accepting even more of a risk when you get in your car to go to work. You’re a billion times more likely to die in a car accident than a terror attack. So why aren’t we campaigning against ridiculous bullshit by insurance companies and state legislations that waste your tax dollars to make them money instead of fixing problems with automobile safety?

It doesn’t sell papers.

Update: It now appears that some people with some experience in Chemistry have questioned the plausability of the terrorist plot.
Update 2: More information about the acquisition of the information that led to the arrests and wide spread media terrorism.

We live in a world of malware. Spyware, Adware, Virii, and generally annoying programs saturate the landscape of the web. Users don’t even have to really try to get these infections either. Just visiting some websites can lead to infection if you’re blissfully unaware of the evil in EULA’s. To combat this problem, a large number of corporations automatically remove Administrator rights from user’s computers. This sounds like a great idea, but atleast until Vista, Microsoft Employees all have administrative rights on their own workstations!

Worse than your IT Staff being unfamiliar with userland without privileges, is the OS DEVELOPER being relatively unfamiliar with its interface without administrative privileges. Even with this protection, attackers can still escalate privileges, or circumvent that fact by exploiting programs that HAVE to run as administrator. Take a look at the Secunia database for JUST the Operating System flaws in Microsoft Windows XP Pro. A search for “antivirus” on Secunia is also a bit depressing, listing 88 vulnerabilities for Antivirus Suites.

Aside from hackers, technologically inclined staff can undermine your group policies in several interesting ways. Network security can be circumvented just as easily. The advent of portable applications and network anonimizers, techniques used by “bad guys” for years, have destroyed policy’s strong hold on the corporate network. Determined users will knock down any and all technological barriers to their productivity.
Even if you’ve managed to take all the precautions to prevent the circumvention of your policies, including BIOS Protection, a determined user armed with google can circumvent your BIOS password and just boot up into Ubuntu or any number of other free, live CD distributions of linux to escape all of your fancy Active Directory Based security policies. From there it’s trivial for users to do what they want. Using WINE and OpenOffice, they can be just as productive as normal users, and far less restricted.

The bottom line is that the users have to be able to comfortably work within your organization with your security policies before your policies are effective at preventing breaches. There are a number of factors, far beyond the reach of most corporate IT Policies & Procedures documents that need to be addressed.

Employees honestly need to feel like a part of the organization, which is difficult when you consider how upper management is distancing itself from the worker bees. As retarded as it sounds, when the average worker is being degraded by executives who take home more in BONUSES than most DEPARTMENTS take home cumulatively in their organization over the course of a year, there’s incentive for corporate espionage and sabotage. This has been witnessed several times in history. I’ll stop before I get political, but bottom line, is there will never be “Information Security” in a country where there’s an obsurd distinction between rich & poor.

Employees must also be given certain amount of Trust to give them a feeling of belonging or exclusivity. The tools to provide accountability to actions on your network are readily available, so should they fall out of line, you can casually remind them or adjust your policy if necessary. If there’s no trust in the organization, the employee is forced to look out exclusively for themselves, which means they’ll be much more likely to act without regard to their impact on the organization.

Employees need room to grow and learn. Without the potential to better themselves and their monetary compensation for their laborious contributions to your organization, the employees will leave, taking with them knowledge of your security measures. They will also be more receptive to ideas of subterfuge, infidelity, and mutiny. Reward your employees whenever possible.

Eventually you’ll get to the IT side of Network Security. Basic preventive and passive monitoring measures should be deployed on the network to prevent outsiders from attacking. The prevention of “insider attacks” requires more than just a Booklet of IT Security Rules. Those rules should be flexible enough to be deployed throughout the organization, with as few exceptions as possible. Ideally you want your computer systems working for you, not getting in your way.

Regardless of the policies you decide on, the IT group should be the FIRST group to adopt the policy, trickle out from there. That way you can determine and fix potential problem for power users before the CTO kicks your door in.

This concept is key to PROGRAMMING. What I find astounding, is a large majority of corporations are adopting this practice for ALL IT related issues, and it’s even saturating into HR and other areas of employment. Working as a Security Administrator, I’m surprised that most employers have decided to not trust their employees. If you can’t trust them, then why would you hire them?

Some key differences between “users” and “employees”. We’ll assume for the sake of argument, that we’re talking about a Web Application and an Employee’s Desktop Computer.

Web Applications:

• Usually allow most of the internet to establish a connection.
• Usually implement a custom or home-grown authentication schema.
• Usually implement a custom separation of privilege system.
• Usually users are not screened prior to access.

Employee’s Desktop Computer:

• Usually require physical acces (normally, badged entrance to a building, sector of a building, and possibly a room key).
• Usually sit behind fairly restrictive firewalls that block unrequested inbound communication from external places.
• Usually implement a centrally controlled authentication system like ActiveDirectory, or LDAP.
• Usually this process is linked directly into HR’s New Hire / Termination Process
• Permission and ACL system’s are usually tied directly into ActiveDirectory and/or LDAP
• Users are screened through the interview process. They also tend to be known to the organization.
• Actions on the systems usually include a system for accountability wherein an event can be traced directly back to a particular user.

Yes, there are exceptions. I know Kevin Mitnik would just walk into a building behind an employee, pretend like he belonged there and sit down at an unused computer and “hack” internally. However, people like Mitnik are exceptions to the rule. Most of these pimply faced, angst ridden, EMO listening script kiddies don’t have the courage necessary to “hack” at a social level.

So why doesn’t your organization trust you? They can easily punish & revoke acccess after repeat offenses. Theoretically, it’s not more work than is currently being done. Actually, if users had administrative rights over their pc’s, they could install the software they need to get their jobs done without putting in tickets to a corporate help desk. Would machines get thrashed by malware and stupid ass HotBar installs? Of course, but how many untrusting environments currently deal with those problems as is?

The fact is, virus and malware writers are clever. Certain processes run as administrator on a windows machine regardless of the user logged in. Using the builtin messaging systems, the malware writers can force their installer to run as administrator if you have an Antivirus process running. So in a sense, we have policies that impact and impede employees while not really eliminating the serious threats they’re being flagged as preventing.

Currently, using a combination of open source tools at work, we’re trusting our users. If they’re not productive, they don’t stick around. We get the IT overhead the hell out of their way and let them be productive. The result has been more effective employees. We do have problems occassionally, but every IT section fights the occassional virus or malware outbreak. Even cooler, the system we’ve adapted has helped us automate a lot of the fight because we’ve had far more time free to implement proactive and reactive network security policies since we’re not spending all our time installing Adobe Acrobat on all 800 of our users’ Desktops.

We’ve also noticed that when users feel trusted, they tend to have a much more positive outlook on the whole IT field. I’ve been in environments where users hate their computer so much they become beligerent the second they get an error message. Granted, we still have angry users, but much less frequently than previously. We hired our employees because they were the best candidates and part of their job is being responsible. If they’re not responsible, they don’t last long.

Of course, there’s an additional piece to consider. Now that we’ve nailed down the monitoring and accountability, we’ve noticed that after users get warned about something once, they generally don’t repeat offenses. They genuinely want to be secure. Do you honestly think your employees want to compromise their personal data, trade secrets, or customer data? Hell no! That’s bad. No one wants bad. They generally don’t know not to click the f*cking monkey until you tell them not to. It’s education.

The internet is a scary place filled with promises of riches beyond your wildest imagination. That promise, techies know is no different than any opportunity that existed prior to the internet. Usually, if it sounds too good to be true, it generally is. Users need and want to be better educated about the threats they face online. Just like you paid that consultant to come in and teach best practices to your programmers, you should put together classes for users to get education on the internet and computers.

If you don’t believe me, I dare you to put together an introductory course to internet safety for your users. Offer the class, don’t force it down their throats and see what the response is. Also, please don’t be ignorant to non-work related issues. Your employees screw around at work, and if they like you, they work at home. So, address clicking on the monkey and myspace.com and the threats that they face on those sites. Don’t be arrogant and make the class fun.

Even if a small percentage come to the first class, they’ll generally spread that knowledge to co-workers and friends virally. The average person wants to know and use best practices for maintaining security on their home & work PCs. They don’t want the world to know that they just bought Yanni tickets!

Generally speaking, not trusting the users is a GOOD thing for PROGRAMMING. However, used as blanket policy for your employees, it creates an environment of distrust and disdain. It will undermine any “team building” seminars you just paid for to help people “synergize”. They’re your employees and many of you spend more time at work than with your own families. If you’re around people you can’t trust, GET OUT OF THERE NOW. IT Policies will not help you in this case.