ElasticSearch CLI Tools - Part 1

2019-05-18T00:00:00+00:00
While working at Booking.com, I was looking for a solution to logging that matched the ease of use and power as Graphite</a> did for metrics. Reluctant to bring a new technology into production, I talked to co-workers and one mentioned that they were using ElasticSearch</a> in some front-end systems for search and disambiguation. He mentioned hearing there were a few projects using ElasticSearch for storing log data.
This began my love-hate-love relationship with ElasticSearch. I've spent the past 8 years working with ElasticSearch professionally and in my spare time. Graphite and ElasticSearch are two projects that change the game in terms of exploring your data. The countless insights I've gained into system performance, application performance, and system and network security with these tools is unparalleled. Tools like Grafana</a> and Kibana</a> allow you to visualize your data quickly and beautifully. As a system and security engineer, sometimes this isn't enough. I spend most of my day in a terminal and needed something to explore and pivot through the data there.
This is the first part, in a many part series about a tool I created to make ElasticSearch's powerful search interface more accessible from the terminal. This tool has been essential to nearly every incident I've investigated. It was developed with the help, patience, and amazing ideas from co-workers both at Booking.com and now at Craigslist</a>.
Perl Setup</h2>
I'm a Perl</a> programmer. You may have strong feelings about that, but Perl has been good to me. The freedom to write code as beautifully, or as ugly, as I need to get the job done is liberating. I recommend using Perl 5.28 or newer with Perlbrew</a>.
You should be comfortable with the command line, so follow the steps to install Perlbrew from it's homepage. After that:
$ perlbrew init
</span>$ perlbrew install -j 8 -n --thread 5.28.2
</span>$ perlbrew switch 5.28.2
</span>$ perlbrew install-cpanm
</span></code></pre>
Now that you have a working, local, user managed Perl, we'll install the
toolset.</p>
$ cpanm App::ElasticSearch::Utilities
</span></code></pre>
The utilities and their dependencies will be installed in your local,
user managed Perl path.</p>
(Some) Utilities Installed</h2>

es-alias-manager.pl</code> - Alternative to
curator</a>
for managing aliases for indexes</li>
es-apply-settings.pl</code> - Applies settings to an index based on index name
and age.</li>
es-copy-index.pl</code> - Tool for copying all (or based on a search) documents
from an index on the same or a different cluster to another index,
optionally supports supplying alternate settings/mappings for the
destination index if it's being created</li>
es-daily-index-maintenance.pl</code> - Alternative to
curator</a>
for maintaining index life spans</li>
es-graphite-dynamic.pl</code> - Script to extract ElasticSearch Performance
metrics into Graphite</a> directly or via
collectd/diamond.</li>
es-status.pl</code> - A quick "how's the cluster" status overview</li>
es-storage-overview.pl</code> - Check how much storage each node and/or index is
consuming in the cluster.</li>
</ul>
And finally, the tool I'm going to be talking about: es-search.pl</code>.  This is
a tool designed with the UNIX philosophy in mind to enable workflows where the
output of one query can be fed into another.</p>
Configuration</h2>
In order to ensure we have the most fun with the tool, let's setup some
defaults to make our command lines shorter.  All of the scripts (and if you're
so inclined, the entirety of the App::ElasticSearch::Utilities</code> functions)
use this config file to determine how to find, connect, and talk to your
ElasticSearch cluster.</p>
Create ~/.es-utils.yaml</code> file with something like this:</p>
---
</span>host</span>: </span>localhost
</span>port</span>: </span>9200
</span>base</span>: </span>syslog
</span>days</span>: </span>1
</span>timestamp</span>: </span>'@timestamp'
</span></code></pre>

host</code> - The host of the hostname or IP of the node you'd like to use to
connect, default is localhost</strong></li>
port</code> - The port to use to connect, the default is 9200</strong></li>
base</code> - Default index base name, defaults to logstash</strong></li>
days</code> - Default number of days to search, defaults to 7</strong></li>
timestamp</code> - Default name of the field containing the timestamp for logging
events, defaults to @timestamp</strong></li>
</ul>
Index Bases</h3>
The idea behind this tool, is to make things as simple as possible.  If you're
like me, you probably use index names to differentiate where shards are
allocated and ultimately, how long shards will exist on your cluster.  On
large indices, where data is variably interesting, I tend to use this pattern.</p>

I want to index HTTP access logs, I'll designate the mappings keying off
the pattern: *-access-*</code></li>
My logs span multiple datacenters, so I'll set allocation rules to make
shards in each datacenter stay</strong> in that datacenter.  If my datacenter tag
is sfo</code>, I'd set a pattern sfo-*</code> to grab those shards</li>
There maybe lower value</em> data in the logs, like requests for images, CSS, or
JavaScript assets.  I want these around, but if they're 90% of my logging
volume and they generally become less interesting more quickly and I'll want
shorter retention rules applied to them. These indexes might include a tag in
the index name of *-bulk-*</code> to make them distinguishable.</li>
</ul>
At the end of this madness I might have a list of indexes like:</p>
</td></td></td></td></td></td>

Index Name</th> Alias</th> Retention</th> Content</th></tr>
</thead>

 ams-access-2019.05.19 </td>  access-2019.05.19 </td>  90d </td>  Normal access logs for `ams` servers </td></tr>
 ams-access-bulk-2019.05.19 </td>  access-2019.05.19 </td>  7d </td>  Uninteresting access logs for `ams` servers </td></tr>
 ams-syslog-2019.05.19 </td>  syslog-2019.05.19 </td>  90d </td>  Syslog data for `ams` servers </td></tr>
 sfo-access-2019.05.19 </td>  access-2019.05.19 </td>  90d </td>  Normal access logs for `sfo` servers </td></tr>
 sfo-access-bulk-2019.05.19 </td>  access-2019.05.19 </td>  7d </td>  Uninteresting access logs for `sfo` servers </td></tr>
 sfo-syslog-2019.05.19 </td>  syslog-2019.05.19 </td>  90d </td>  Syslog data for `sfo` servers </td></tr>
</tbody>
</table>
If I wanted to search those indexes, I could just use --base access</code> as they
all will be parsed to the correct bases.  If you're not sure what
es-search.pl</code> might think of what bases</em> you have available, ask it to tell
you:</p>
$ es-search.pl --bases
</span>Bases available for search:
</span>	access
</span>    ams-access
</span>    ams-access-bulk
</span>    ams-syslog
</span>    sfo-access
</span>    sfo-access-bulk
</span>    sfo-syslog
</span>    syslog
</span># Bases: 8 from a combined 6 indices.
</span></code></pre>
Handling More Than One Index Base with Ease!</h3>
That's all fine and good if all of your indexes contain the same document
types.  That's unlikely as you should be splitting different document types up
into separate indices, if not clusters.  If you want to work with
es-search.pl</code> across all those indexes easily, it will need to know the
correct timestamp field.  To enable per-base timestamp fields, you can just
add a meta</code> section to your ~/.es-utils.yaml</code> file.</p>
---
</span>host</span>: </span>localhost
</span>port</span>: </span>9200
</span>base</span>: </span>syslog
</span>days</span>: </span>1
</span>meta</span>:
</span>  </span>access</span>:
</span>    </span>timestamp</span>: </span>timestamp
</span>  </span>ossec</span>:
</span>    </span>timestamp</span>: </span>ts
</span>  </span>zeek</span>:
</span>    </span>timestamp</span>: </span>event_ts
</span></code></pre>
Now es-search.pl</code> and the rest of the utilities will know that when you
specify --base zeek</code> the timestamp field to sort on will be event_ts</code> and
you won't need to think about adding --timestamp event_ts</code> to the command
line.</p>
Seeing Data</h2>
Now that you're configured, we can just run: </p>
$ es-search.pl
</span>= Querying Indexes: syslog-2019.05.19
</span>---
</span>action: connect
</span>hostname: janus
</span>message: 'connect from unknown[102.165.34.33]'
</span>proc: smtpd
</span>proc_id: 30775
</span>program: postfix/smtpd
</span>src: unknown
</span>src_ip: 102.165.34.33
</span>tags:
</span>  - decoder_syslog
</span>  - mail
</span>  - postfix
</span>timestamp: 2019-05-19T02:07:34.861416
</span>total_time: 0.004363
</span>
</span><snip>
</span>
</span># Search Parameters:
</span>#    {"bool":{}}
</span># Displaying 20 of 357 in 0.0584328174591064 seconds.
</span># Indexes (1 of 1) searched: syslog-2019.05.19
</span></code></pre>
Each document's _source</code> is YAML printed to the screen.  This is not the usual
use case for es-search.pl</code>, so let's do better.  It's also likely that the
documents you're viewing may not contain all the valid fields in the index.</p>
Finding the Fields in the Index</h2>
When you start working with ElasticSearch indexes, you may not know all the
fields available for search.  es-search.pl</code> allows you to explore a bit:</p>
$ es-search.pl --base syslog --fields
</span>Fields available for search:
</span>	- action
</span>	- dev
</span>	- dst_geoip.continent
</span>	- dst_geoip.country
</span>	- dst_geoip.location
</span>	- dst_ip
</span>	- dst_port
</span>	- exe
</span>	- file
</span>	- hostname
</span>	- in_bytes
</span>	- message
</span>	- out_bytes
</span>	- proc
</span>	- proc_id
</span>	- program
</span>	- proto_app
</span>	- rec_id
</span>	- src
</span>	- src_geoip.city
</span>	- src_geoip.continent
</span>	- src_geoip.country
</span>	- src_geoip.location
</span>	- src_geoip.postal_code
</span>	- src_ip
</span>	- src_port
</span>	- src_user
</span>	- tags
</span>	- timestamp
</span>	- timing.phase
</span>	- timing.seconds
</span>	- total_time
</span># Fields: 32 from a combined 1 indices.
</span></code></pre>
This will help you understand what an index contains.  Maybe you wanna see
what's in a field?  There's two ways, the first with search, the second with
aggregations.</p>
Finding Field Values with Search</h3>
The simplest, and least taxing way to ask ElasticSearch what a field contains
is to query the index and return the relevant field.  To optimize for
documents containing the field, we can use the --exists <fieldname></code> filter.</p>
If I just want to see the most recent 20 documents where the field proc</code>
exists and just</em> see the proc</code> entry, it's as simple as:</p>
$ es-search.pl --exists proc --show proc
</span>= Querying Indexes: syslog-2019.05.19
</span>timestamp    proc
</span>2019-05-19T02:04:06.135686    smtpd
</span>2019-05-19T02:04:06.135786    smtpd
</span>2019-05-19T02:04:05.856884    smtpd
</span>2019-05-19T02:03:46.471311    smtpd
</span>2019-05-19T02:03:46.471352    smtpd
</span>2019-05-19T02:03:46.199116    smtpd
</span>2019-05-19T02:03:37.013022    smtpd
</span>2019-05-19T02:03:37.012866    smtpd
</span>2019-05-19T02:03:36.741711    smtpd
</span>2019-05-19T02:03:18.239108    smtpd
</span>2019-05-19T02:03:18.239135    smtpd
</span>2019-05-19T02:03:17.947805    smtpd
</span>2019-05-19T02:03:07.837098    smtpd
</span>2019-05-19T02:03:07.837133    smtpd
</span>2019-05-19T02:03:07.553645    smtpd
</span>2019-05-19T02:03:07.342514    smtpd
</span>2019-05-19T02:03:07.342686    smtpd
</span>2019-05-19T02:03:07.067929    smtpd
</span>2019-05-19T02:02:57.157830    smtpd
</span>2019-05-19T02:02:57.157612    smtpd
</span># Search Parameters:
</span>#    {"bool":{"must":[{"exists":{"field":"proc"}}]}}
</span># Displaying 20 of 85 in 0.0445699691772461 seconds.
</span># Indexes (1 of 1) searched: syslog-2019.05.19
</span></code></pre>
This might not give me the best understanding of what the field is, but
already, I know that postfix</code> log entries are setting this field.</p>
Finding Field Values with Aggregations</h3>
We can do a lot better by leveraging aggregations in ElasticSearch.  To do so,
we ask es-search.pl</code> for the top values.</p>
$ es-search.pl --top proc
</span>= Querying Indexes: syslog-2019.05.19
</span>count    proc
</span>224  smtpd
</span>27   smtps_smtpd
</span>12   qmgr
</span>9    localsmtp_smtpd
</span>6    cleanup
</span>4    submission_smtpd
</span>3    anvil
</span>3    lmtp
</span>3    pipe
</span># Search Parameters:
</span>#    {"bool":{}}
</span># Displaying 9 of 693 in 0.00798892974853516 seconds.
</span># Indexes (1 of 1) searched: syslog-2019.05.19
</span>#
</span># Totals across batch
</span>#
</span>count    proc
</span>224  smtpd
</span>27   smtps_smtpd
</span>12   qmgr
</span>9    localsmtp_smtpd
</span>6    cleanup
</span>4    submission_smtpd
</span>3    anvil
</span>3    pipe
</span>3    lmtp
</span></code></pre>
We now have the top 20 (or fewer if there's not 20 total) values in the proc</code>
field.</p>
Putting It Together</h3>
It looks like proc</code> is the component</em> piece for postfix</code> syslog data.  To be
sure, let's ask ElasticSearch for the top programs with the top 10 procs each.
Since es-search.pl</code> is designed to make this easy, we type almost exactly
that:</p>
$ es-search.pl --top program --with proc:10 --exists proc
</span>= Querying Indexes: syslog-2019.05.19
</span>count  program
</span>224  postfix/smtpd              terms.proc    smtpd    224
</span>35   postfix/smtps/smtpd    	terms.proc    smtps_smtpd    35
</span>12   postfix/qmgr    			terms.proc    qmgr    12
</span>9    postfix/localsmtp/smtpd    terms.proc    localsmtp_smtpd    9
</span>6    postfix/anvil    			terms.proc    anvil    6
</span>6    postfix/cleanup            terms.proc    cleanup    6
</span>6    postfix/submission/smtpd   terms.proc    submission_smtpd    6
</span>3    postfix/lmtp               terms.proc    lmtp    3
</span>3    postfix/pipe               terms.proc    pipe    3
</span># Search Parameters:
</span>#    {"bool":{"must":[{"exists":{"field":"proc"}}]}}
</span># Displaying 9 of 304 in 0.0130970478057861 seconds.
</span># Indexes (1 of 1) searched: syslog-2019.05.19
</span></code></pre>
Let's break down that query:</p>

--top program</code> - Top aggregation, infers terms</code>, uses the value of --size</code> which defaults to 20</em></li>
--with proc:10</code> - Sub aggregation, form is agg_type</strong>:field name</strong>:sub size</strong>

agg_type</code> - defaults to terms and can be omitted, but can also be: significant_terms</code>,max</code>, min</code>, sum</code>, avg</code>, cardinality</code></li>
field_name</code> - is required and is the sub aggregate field name</li>
sub_size</code> - defaults to 3</strong></li>
</ul>
</li>
--exists proc</code> - Filter the entire aggregation to just documents with the proc field</li>
</ul>
Wrapping up for now</h2>
I think this is a reasonable point to pause.  This provides you with enough
information to start getting your feet wet with the tool.  In the next part,
I'll examine building useful queries and how this tool enables pivoting and
data exploration.</p>
If you can't wait til next time, run: es-search.pl --manual</code> to get an in
depth overview of the options available.  See below for that man page online:</p>

GitHub Project Page: reyjrar/es-utils</a>

es-search.pl</code> man page</a></li>
</ul>
</li>
MetaCPAN Project Page: BLHOTSKY/App-ElasticSearch-Utilities</a></li>
</ul>


systemd-resolved is broken
2017-12-20T00:00:00+00:00
Full disclosure, I'm not a fan of systemd.  I started working with Linux in
the late 90's and watched it grow from a marginalized operating system to the
most dominant operating system in the datacenter.  I've lived through so many
"year of the Linux desktop" years I remember when it wasn't a joke.  From my
vantage point, administering Linux servers professionally for nearly 20 years,
systemd is Linux on the desktop at the cost of Linux in the datacenter.</p>
Why do I feel this way? It's mostly the reinvention and incorrect
implementations of core UNIX tools and modalities.  There's a lot of
information on systemd out there.  There's a lot of bias involved.  So, today,
I'm not going to talk about that.  I am going to address a critical mistake in
the systemd-resolved daemon which implements DNS lookups for systems running
systemd.</p>
I'll jump right to the work-around.  If you're running a system which is using
systemd, you should probably be running systemd-resolved configured to use a
single DNS resolver, 127.0.0.1, and run Unbound</a>.
There are resources on how to configure and run Unbound, but the best is
Calomel's Unbound Tutorial</a>. If you
need to maintain consistent, reliable DNS resolution that's compatible with
previous versions of Linux, the only way to do that is to have a single DNS
server in /etc/resolv.conf.</p>
</span>
Why This Matters</h2>
This thread on
systemd-resolved</a> explains the
issue.  Yes, putting external DNS servers into your internal servers
/etc/resolv.conf is not great form, but that's completely missing the point
exposed in this bug report.</p>

systemd-resolved is implementing state tracking against a stateless
protocol.</p>
</blockquote>
Not only that, but it does it poorly.  In the cases described by the
commenters, a temporary blip in connectivity to internal DNS servers wound up
blacklisting them indefinitely.  In my nearly 20 years as a Linux admin, I've
seen nearly every junior admin come up with the same idea after their first
DNS outage, "Why don't we just keep track of what DNS servers respond and then
ignore ones that are failing?"  It sounds great, but because DNS is a
stateless protocol by design, determining "working server" from "not working
server" is profoundly more difficult then issuing a HTTP request to a status
handler.  It's complicated.</p>
The Old Behavior</h2>
So, there's a lot of misconceptions about glibc's resolver library, so I'm
hoping to squash a bit of that and address how</em> most name resolutions work on
most Linux systems.  Yes, it's possible to use a different resolver library
and those libraries may not implement resolution the same.  However, I want to
talk about the glibc resolver and how it interacts with /etc/resolv.conf on a
stock CentOS 6 system and every UNIX and Linux prior.</p>
Here's a sample /etc/resolv.conf</p>
search edgeofsanity.net
</span>nameserver 192.168.1.1
</span>nameserver 9.9.9.9
</span>nameserver 8.8.8.8
</span></code></pre>
First things first, /etc/resolv.conf supports only three nameservers, and
further servers are ignored.  I've seen up to eight servers in resolv.conf's
administered by experienced, knowledgeable folks. Remember, only the first
three are ever queried.</p>
So what happens with this resolv.conf?  Well, if 192.168.1.1 is responding to
queries, it will always be used to resolve every query.  If a query passes the
timeout, the default is 5 seconds, without a response, the query will be
resent to 192.168.1.1 once more before advancing to 9.9.9.9.  These counters
are tracked internally by the process</strong> running the resolver library.  These
are not global counters, they are local to each process</strong>.  This particular
failure case is also per-query</strong>, meaning each DNS query will have to
timeout twice to 192.168.1.1 before advancing to 9.9.9.9.</p>
Why? Well, a timeout could happen for any number of reasons.  A timeout of a
nameserver for one query doesn't predict a timeout in the future to the same
server for the same query.  It's complicated.</p>
What this configuration guarantees is that every query will take at least 10
seconds to resolve if 192.168.1.1 is down.  This is less than ideal, so we can
improve</em> that a little by adding options.</p>
search edgeofsanity.net
</span>nameserver 192.168.1.1
</span>nameserver 9.9.9.9
</span>nameserver 8.8.8.8
</span>options timeout 1 attempts 1
</span></code></pre>
By setting timeout</code> to 1 second and attempts</code> to 1, we'll try 9.9.9.9 if
192.168.1.1 doesn't respond within 1 second.  Again, this is per-query,
per-process, so every query will always try 192.168.1.1 before moving on to
9.9.9.9, because, repeat after me, "a timeout of a single query to single DNS
server cannot predict that even the same query to the same server will timeout
at any point in the future."</p>
This improves</em> the failure case for 192.168.1.1 becoming unavailable, but
it's still 1+ second for every DNS query, which is unacceptably slow for any
web-scale service.  There's another option we can introduce to decrease the
impact the DNS server being unavailable has on our servers:</p>
search edgeofsanity.net
</span>nameserver 192.168.1.1
</span>nameserver 9.9.9.9
</span>nameserver 8.8.8.8
</span>options timeout 1 attempts 1 rotate
</span></code></pre>
We introduce the rotate</code> option to the config file.  If you were to run this:
while true; do getent hosts www.google.com; done</code> You'd probably be surprised
to see EVERY</strong> query going to 192.168.1.1.  Maybe you can guess why that is?
That's right! The rotate</code> option is per-process, so each time we run getent</code>
we start a new process, which starts at the first name server for the first
query and continues on to the next server for the next query.  Failures
per-query are still processed the same way.</p>
If you had a failure of 192.168.1.1, you'd have more than 33% of DNS queries
taking 1+ seconds to resolve.  Why? Again, rotate</code> is per-process so long
running processes will rotate through the bad server every 3 queries.
However, every new process will always start at the beginning of the list.</p>
The New Behavior</h2>
OK, so what's described in the GitHub issue is the systemd-resolved's author
deciding to break a fundamental design in the DNS resolution on UNIX systems.
Servers are never</em> skipped in the previous glibc resolver world.  This is
because, and I'll say it again, a timeout for a single DNS query to a single
DNS server does not predict a timeout for that same query to that same server
at any point in the future. The systemd-resolved behavior now adds this state
to a stateless protocol, which leads to unpredictable and inconsistent
behavior in one of the lowest level, most misunderstood, and most critical
components in your infrastructure.</p>
There is a way to work-around this, if every DNS server in the list of DNS
servers is marked as being problematic, systemd-resolved falls back to the
default behavior of going through every server in the list and resetting their
state.  The easiest way to ensure this happens is to list a single nameserver
in the /etc/resolv.conf settings.  This will force a short circuiting in the
state tracking logic.</p>
In Closing</h2>
I'm not going to bash systemd or any of it's authors or maintainers.  They're
doing their best to solve hard problems.  I do disagree fundamentally with
their direction and assumptions, but they're writing code and dealing with
angry communities, and I won't pile on.  However, this behavior is
fundamentally different than everything else in the space and represents what
I fear is a naivety and disinterest in understanding the problem space.  If
you administer Linux systems professionally, you need to be aware of this
difference and how it will impact your infrastructure if there are issues with
upstream DNS providers.</p>
It's entirely possible this change in behavior will have no or very little
impact on your infrastructure.  It's important to understand this difference
as DNS is often impacted by or impacting the availability of your services.</p>
Updates</h2>
First, I got something wrong.  In the case we rotate</code> enabled and 3
nameservers, approximately 50% of queries will take 1+ seconds to resolve.
This is because the state isn't magic, it's a simple pointer that's
incremented each time.  Consider, query #1 goes to 192.168.1.1, it times out,
the pointer is advanced to 9.9.9.9 and it succeeds.  Query #2 comes in and
that pointer is advanced to 8.8.8.8, it succeeds.  Query #3 comes in, the
pointer is advanced to 192.168.1.1 it times out and moves on to 9.9.9.9.
Rinse and repeat.</p>
Laurent Bigonville</a>
suggested removing resolve</code> from /etc/nsswitch.conf.</p>
Paul Vixie</a> noted that application developers
should consider using getdns</a> in their applications
as it's a modern, smart resolver library.</p>
Weronika Pawlak graciously translated this post into
Finnish</a>.</p>


VPNs and Internet Privacy
2017-07-16T00:00:00+00:00
After getting a few questions from concerned folks about VPN services. I
realized this might be better served as an article. This way anyone who is
curious about how to protect themselves better online can reference it.</p>
The Bad News</h3>
Well, there's really no easy way to this: There is very little, if any,
privacy on the Internet.</strong>  Even after following all of the advice I'm about
to give, all sorts of clever folks in the Valley and beyond are envisioning
clever</em> new ways to improve the "User Experience" (UX) and in the process
accidentally creating newer, clever means to circumvent any and all privacy
controls you might deploy.</p>
</span>
This never used to bother me all that much.  I worked at one of the largest
e-commerce sites in the world.  I routinely used meta-data to piece together
stories about interactions with our site that literally scared me, both because
of what someone had attempted, and because I was able to replicate it with so
very little data.  I am beyond</strong> amused that Donald Trump signing S.J. Res
34</a>
into law is getting so much attention.  The Snowden
leaks</a> were MUCH</strong> more
terrifying than this legislation, and spoiler alert</em>, your ISP was already
tracking you and making money off your habits.</p>
I Came Here For The VPNs Smart Ass</h3>
Right, so you came here for advice on which VPN service to use.  There have
been lots of opinion pieces already written on this, so feel free to search
DuckDuckGo</a> for the "best VPNs of 2017."  The truth
is, a lot of these VPN services are probably much worse than your ISP, who for
all their short comings is a legitimate, accountable business.  This is my
first piece of advice:</p>
Step 1: Choose your ISP wisely</h2>
Yes, Comcast and Verizon have the fastest speeds.  They don't rank so well in
privacy.  If that's important to you, vote with your business</strong>.  The EFF
Who has your
back?</a>
reports the ISPs with the best reputations on Privacy concerns.  I am lucky to
live in a Sonic.net</a> coverage area.  Find small, local
ISPs near you and talk to them about their privacy policies.  They'll love to
talk you about those things to get the word around.</p>
VPNs Aren't Private in the Way You Think They Are</h3>
You'll hear some privacy advocates recommend against</strong> using VPN services in
the US. It's likely using VPN services in any US allied nation is probably
worse</em> than using your own ISP.  Why? Allies share intelligence.  OK, I'll
say that again: Allies share intelligence.  It may still be illegal for the US
to spy on US citizens, but it's not illegal for the UK, Australia, New
Zealand, and Canada to spy on US citizens and share anything they find with
the US Government in exchange for the same favor.  Using VPN services in
countries not allied with the US could be safer from the point of view of
access by the US government to your traffic, but it's likely to raise flags
and cause you to be under more scrutiny from the government anyways.</p>
All things considered, you probably want to use a VPN service operating under
the laws of the country you reside. If the reason you're using the service is
to circumvent laws in the country where you reside, this article probably
isn't for you.  None of my recommendations will protect you from yourself and
none of it will save you from prosecution.  I wish it could, but that's not
the Internet we</strong> built.</p>
What is a VPN?</h2>
First you have to know the Internet is just a large mesh of computers, each with
connectivity to one or more computers.  These computers agree to transmit data
for one another to end points they're not directly connected.  There's lots of
math and physics involved.  For our purposes, just think of it as a socialist
group of computers sending data down a path towards its destination.  There's
some meta-data wrapped around the data to help the computers determine where the
traffic is from and where it might need to go next.  There's a constant exchange
between all the computers informing one another to which computers they can
transmit data.</p>
There's a lot of trust in this system, and up until recently, most of the data
itself was unobstructed. Think of it as sending postcards.  Everyone could see
the source, destination, and the entire message. If they carried it, they could
modify it without much ado.  Since the Snowden leaks, the Internet has gotten
serious about encryption.  This means there are more coded messages flying
around, but it's still on the back of postcards and anyone in the middle can
read the source and destination.</p>
So, now VPN's.  You probably had to use a VPN at work at some point to access
internal company resources while traveling or working from home.  A Virtual
Private Network</em> uses the Internet to allow two computers that are not next to
each on their physical network</em> behave as though they were on the same physical
network.  They usually encrypt the traffic at a very low level to prevent
computers in between from knowing the actual source or destination of the
traffic.</p>
This is good for making internal resources accessible to employees from anywhere
in the world, but it doesn't exactly gain you much in the way of privacy.  At
the VPN service, all of your data is unwrapped and shipped off over the Internet
to it's destination.  You can transmit encrypted traffic, such as HTTPS, inside
a VPN and the VPN service won't know anything about the content of the message,
but they will still know it's source and destination.</p>
So, they won't know what was in the content, but they will know that after
loading an advertisement that your browser prefetched DNS records for
"erectile-dysfunction.com" you then created an encrypted connection to
"erectile-dysfunction.com" and loaded not one, but several pages, including one
that called out to "verified-by-visa.com."</p>
What is DNS?</h2>
Computers prefer numbers, humans prefers letters and words.  To resolve this
issue, (haha, you see what I did there? Don't worry if you don't, I'm laughing
by myself</em>) we have the Domain Name System</strong> or DNS</strong>.  DNS allows you to
type: "www.google.com" and your computer knows it needs to send data to
"172.217.5.100" or whatever your DNS server says is www.google.com.  It sounds
simple, but it's a system with implied trust and some really cool tricks.  This
makes DNS an incredibly complex topic, so I'll gloss over all the technical
details and say it does this pretty well, and it does so in plain-text.  Anyone
who sits between you and your local DNS resolver can see every name your
computer tries to resolve without any obstruction, again like a postcard.</p>
Almost 100% of DNS traffic is unencrypted today.
DNSCrypt</a> is looking to change that, but it's just
now gaining traction and you probably aren't using it.</p>
OMG, Why Do I Care?</h3>
Great question! Web pages started out pretty simple, but included ways to link
and even include content from other web sites on your page.  Over time, things
got much more complex, and data gets loaded from everywhere.  The average web
page loads over 100 assets and almost 2 megabytes of
files</a>.  This
is a lot of data and a lot of network requests.  To keep that in perspective,
the original Legend of Zelda</a>
came in around ~128 kilobytes, so your average web page is using about 15</strong>
Legend of Zeldas</em> worth of mostly garbage.</p>
Since we all love speed and browsers are throttled by your home internet
speed, a clever person in the Valley or beyond came up with an idea to improve
the User Experience.  They realized, once you load the page, you read it, your
browser is sitting by idly waiting for the next command.  Most of the time,
the next command involves clicking a link, so resolving the hostname, fetching
the page, reading the page for any assets it needs, resolving more names,
fetching those objects.  It takes time and you have to issue those first
requests to know how many of the supporting requests are required.</p>
So, this engineer thought, "what if we scan the page you're reading for links
while you're reading and start that process?"  And that's what they did.  Most
modern browsers use "content pre-fetching" to give you an artificially fast internet
connection.</p>
Step 2: Disable Prefetching</h2>
The terrible truth about prefetching is anyone who can see only your DNS
requests</em> can probably reverse engineer the words you just typed into Google's
Search Bar REGARDLESS</strong> of whether or not you are using HTTPS.  Here's a paper
from 2010 detailing the privacy implications of DNS
prefetching</a>.</p>
Firefox (All)</h3>

Open a new tab</li>
Type: about:config</li>
Agree you're breaking your warranty</li>
In the search bar, type: network.dns.disablePrefetch</li>
If the value is "false", double-click on it to change it to "true"</li>
</ol>
Google Chrome (Mac)</h3>

Select "Preferences" from the menu bar</li>
Select "Advanced"</li>
Disable the following:</li>
</ol>

Use a web service to help resolve errors</li>
Use a prediction service to help complete searches and URLs from the URL bar</li>
Use a prediction service to load pages more quickly</li>
</ul>
Safari (Mac)</h3>
From Apple Discussions</a>:</p>

Open Terminal</li>
defaults write com.apple.safari WebKitDNSPrefetchingEnabled -boolean false</li>
Restart Safari</li>
</ol>
Step 3: Tweak Your Browser Privacy Settings</h2>
I won't go into details on these, but search
DuckDuckGo</a> for how to do each on your browser.</p>

Disable Usage Reporting</strong> - Reports to the browser developers how you use
their product.</li>
Disable Crash Reporting</strong> - Reports crashes, including potentially
sensitive or confidential information to the browser developers when your
browser crashes.</li>
Disable WebRTC</strong> - WebRTC allows advertisers to fingerprint you by getting
information about your home network.</li>
Disable Cookies (Third Party)</strong> - Completely disabling cookies will prevent
most sites from working, but this limits it to just sites you visit.</li>
Uninstall Flash and Java</strong> - It's 2017, these two need to GTFO
of our browsers. I don't have enough curse words to describe why.</li>
</ul>
Step 4: Extend Your Browser</h2>
To enhance your privacy everywhere, regardless of whether you're using a VPN
or not, there are some tools freely available.</p>
Ad-Blocking</h3>
I work in IT Security. I know, "ad blocking takes revenue away from the little
guys."  Unfortunately, the reality is most ad networks do a sub-optimal job of
curating their content.  It's not uncommon for malware or viruses to be served
via a legitimate ad network.  Even if that weren't the case, the advertiser
and the content-provider may have a contract in place with privacy clauses,
but their contract doesn't extend to you, the casual web surfer.  So, you
may trust a particular website, but that doesn't mean you necessarily trust
all of their ad partners.</p>
For privacy and security reasons, you need to be using an ad-blocker.  The
best of the breed is uBlock Origin</a>.  It's
light weight, efficient, and super configurable.  Be sure to dig into the
options and go nuts with block lists.</p>
Privacy Enhancements</h3>
The EFF publishes two incredibly useful extensions:
HTTPSEverywhere</a> and
PrivacyBadger</a>.  HTTPSEverywhere preloads
a new browser feature that prevents snooping and traffic interception by
forcing all communication from popular sites to be HTTPS.  PrivacyBadger is
the EFF's curated list of privacy threats.  It's a good combo to compliment
the community lists from uBlock Origin.</p>
As I mentioned, web pages these days load resources from all over the place.
There's a number of common libraries used in websites that are loaded from
"Content Delivery Networks" (CDNs).  These CDN's wind up seeing almost
everything you do on the internet because they receive the context of where
these assets are loaded.  CDN's are generally seen as good because they speed
up the internet by having bigger, faster connections in more places than most
content creators can reasonably afford.  They come at cost to privacy though,
as they see what you're doing across thousands, if not millions, of sites.</p>
Enter Decentraleyes</a>. It contains most
of those common libraries locally. When it can serve the library from a
version it has loaded on you computer, it injects the library locally instead
of fetching it from the CDN.  This dramatically reduces the amount of network
usage and the number of connections per site loaded.</p>
If you happen to be on Firefox</a>, and you probably
should be, there's another extension called
BetterPrivacy</a>.
This extension helps clear out persistent tracking data via configurable
thresholds.  You can configure it to wipe them every time you close the
browser, or every X minutes.</p>
Link and Click Tracking</h3>
Do you ever read the full links people post these days?  Or held your mouse
over a link on a Google search results.  There's a lot of extra junk in those
links, often with a utm_</code> prefix.  This contains information about the ad
campaign, the medium, and in the case of mobile devices, the application
information used to view the link.  The thing is, these links will work the
same without all of that crap.  I use
PureURL</a>
to strip unnecessary parameters from links to prevent leaking sensitive data
from my devices.</p>
Again, if you're running Firefox, there's an additional extension I recommend,
GooglePrivacy</a>
which does the same thing  PureURL does, but your Google's search results and
their specific internal link tracking.</p>
Paranoid Mode</h3>
For extra paranoid, I recommend the following extensions, but keep in mind,
most of the internet stops working without you explicitly whitelisting
resources in one or more these utilities:</p>

NoScript</a> - Disables JavaScript loading, provides
boundary enforcement, prevents Click
Jacking</a></li>
uMatrix</a> -
LittleSnitch</a> for
everything your browser does, everything.</li>
CertificatePatrol</a> -
Reports when the security certificate for a site changes.</li>
</ul>
Step 5: Understand "Private Browsing"</h2>
"But wait," you interject, "Why go through this madness when my browser has
private browsing mode?"  Excellent question!  Private browsing is more
"private from other users on my computer" than "privacy from the government."</p>
When you start a private browsing session, the short and long term caching and
storage for your browser are pointed to a new, unique location for as long as
the private browser window is open.  Think of it as changing your clothes, but
grabbing a different wallet, with different IDs and credit cards each session.</p>
That sounds</em> good, but you don't change your internet address, and you're not
changing your DNS servers.  This means, to your ISP or anyone able to see your
network traffic, you're still you.  So, from an ISP or government perspective,
nothing's changed.</p>
Private browsing disables history tracking and makes you look like a new user
to the websites you're visiting, but that's about it.  Another nice feature is
the local cache is cleared when you close the window.  So all those
embarrassing pictures of the Icy Hot Stuntaz won't be sitting around for your
loved ones to find.</p>
Step 6: VPN Up</h2>
OK, at this point, you've locked your browsers down, but you may have a few
good reasons to use a VPN:</p>

You're going to be using a network you don't trust, like mobile networks or
other people's WiFi.</li>
You're on a mobile device. Mobile networks are terrible for privacy and
there aren't many choices aside from VPNs.</li>
You're in a country like China where your communication is severely
hindered by your local government.</li>
</ol>
Those are the only reasons most responsibly savvy users may find they need a
VPN.  As I stated before, I'd recommend choosing a VPN service that operates
legally in your own country.  Some VPN service providers operate in many
countries and all you to route our traffic to any one of those countries.  If
given the option, even if you live in the USA, I'd recommend using a VPN end
point in your country.  The exception to this being use-case #3.  In those
instances, choose a country that's least likely to cooperate with your
government, but understand that you may wind up drawing attention from your
local government in doing so.</p>
My preference is to setup your own VPN server by building a
pfSense</a> image on Microsoft Azure or Amazon's AWS.  The
WebUI for pfSense if pretty easy to configure your VPN Server, even export
profiles to use in your devices.</p>
If that's too technical for you, I'd look into VPN providers with a history of
transparency who are vocal and active with their privacy protections.  Some
signs a VPN Provider takes privacy seriously:</p>

Zero logging policy - This is almost always a lie as logs are necessary for
support, so ask about what is logged and how long it stays on disk.</li>
Warrant Canary - The government doesn't allow companies to advertise when a
warrant has been served.  However, librarians came up with a clever system
called a "warrant canary."  It's a notice posted stating "No warrants have
been served in the past X days."  When a warrant is served, that posting is
removed.  It's a legal grey area and any service that takes privacy
seriously will have one.</li>
Accept BitCoin for payment.  I don't have enough time to talk about BitCoin
here, but if you want to protect your billing information, you need to use
BitCoin.</li>
Disclosure of the laws they operate under and where there servers are
physically located.</li>
Do they provide DNS service?  If not, then you're still leaking data over
your ISP.</li>
Disclosure of all third party services.  Who do they use to send you mail
when your bill is due?  Are they using a third party solutions for
monitoring, instrumentation, or operations?</li>
</ol>
I wish I could tell you, "use this service," but it's not that simple.
There's a lot to consider when making this choice and the answer depends on
your expectations and your comfort levels.  I believe the only way to be
certain is to run your own VPN service, but with the rest of the tooling I
mentioned in place to protect you.</p>
Abandon all hope..</h2>
I probably bummed you out.  I'm not sorry.  I've been watching privacy erode
on the internet for the last 20 years.  It's hard to do privacy right on the
internet.  You're often an accidental mouse-click away from blowing all
your protections.  Hopefully this helps you navigate a bit smarter and
understand a bit better that anyone purporting to sell you privacy on the
internet is just blowing smoke up your ass.</p>
Stay safe, my friends.</p>
Translations</h2>

Karolin Lohmus graciously translated this article into Estonian</a>.</li>
Artur Weber & Adelina Domingos graciously translated this article into Portuguese</a>.</li>
</ul>


In which he authors a book on OSSEC
2013-08-04T00:00:00+00:00
In 2004, when I was starting a new job at the National Institute on
Aging's Intramural Research Program</a> I began
evaluating products to meet
FISMA</a> requirements for
file integrity monitoring.  We already purchased a copy of Tripwire, but I
was being driven mad by the volume of alerting from the system.  I wanted
something open source.  I wanted something that would save me time, rather
than waste 2 hours a day clicking through a GUI confirming file changes
caused by system updates and daily operations.</p>
At the time, I found two projects:
Samhain</a> and
OSSEC-HIDS</a>.  Samhain is a great project that does
one thing and does that one thing very well.  However, I was buried in a
mountain of FISMA compliance requirements and OSSEC offered more than file
integrity monitoring; OSSEC offered a framework for distributed analysis of
logs, file changes, and other anomalous events in the same open source
project.</p>
I now work at Booking.com</a> and manage one of the
world's largest distributions of OSSEC-HIDS.  My team and I are active
contributors to the OSSEC Community.  After nearly a decade of experience
deploying, managing, and extracting value from OSSEC, I was approached to
write a book introducing new users to OSSEC.  After 6 months of work, the
book has been published!</p>
Instant OSSEC Host-based Intrusion
Detection</a></p>
</span>
Writing a book is hard, kids.</h2>
I write a lot.  I thought, "writing a book would be fun and easy!"  And, in
some ways it is.   It's also very challenging writing for an editor.  It's
also very humbling to receive feedback on your book from the technical
reviewers.  Most of my writing, never makes it anywhere public.  Even the
writing that ends up on my blog generates very little feedback.  Now,
imagine someone scrutinizing every single technical detail 50 pages of your
writing.  Every sentence fact checked and marked up.  Every mistake you made
highlighted and commented by multiple technical reviewers.  That's hard.</p>
It doesn't stop there.  From the first draft, the editor points out every
instance of awkward wording or incorrect grammar.  Once your editors and
technical reviewers are satisfied, the book moves on to the Technical
Editor.  This person is responsible for ensuring language consistency and
compliance to the conventions of the dialect of which the book is written.
If you're not comfortable with your work being torn apart and reassembled,
writing a book may not be for you.  It was grueling at times.</p>
It does help to have a good group of people involved in the process.  I was
thankful to have a great team of editors and reviewers for this book at
Packt Publishing</a>.  Now, that the job is done
(side note: the job is never done</em>), I am grateful to have been given this
opportunity.  It was a very rewarding experience, and like all rewarding
experiences, it wasn't easy.</p>
OSSEC-HIDS and so can you!</h2>
So, what's this OSSEC-HIDS this book is all about?  I, like most of you,
struggle with putting my logging data to good use.  I have several posts on
this blog dedicated to extracting value from logging.  I also have
compliance requirements, previously FISMA, currently PCI-DSS.  OSSEC is an
open source security project which incorporates a number of useful features
that make it worth a look.</p>

Distributed log analysis</li>
File integrity monitoring</li>
Rootkit detection</li>
Policy auditing</li>
Alerting and active response system</li>
</ul>
All of these features interoperate in the same event system, providing
richer context in an infinitely customizable environment.</p>
Distributed log analysis</h3>
OSSEC runs best in a client/server model.  You can have hybrid deployments
to aggregate and analyze events at different points in your network.  The
clients run as agents, propagating log data to the servers.  The servers
than evaluate the log data using decoders to extract data and rules to
analyze the data.</p>
OSSEC ships with an impressive list of decoders and rules capable of
analyzing and classifying events from standard system services including,
but not limited to : ssh, ftp, email, web servers, LDAP, ActiveDirectory,
Windows EventLogs, Windows Registry, Cisco PIX, Juniper NetScreens, and most
Linux/BSD/Solaris system daemons and kernel messaging.</p>
The OSSEC decoders and rules are
extensible</a>,
so if your use case isn't addressed it's easy to add.</p>
File integrity monitoring</h3>
If you work in IT security, you hate file integrity monitoring.  It's
because of products like Tripwire, which do file integrity monitoring, but
are so loud and obnoxious about it, that the value is drowned in a sea of
unending harassment of correct, but uninteresting events.  I desperately
wanted out of what I found largely unactionable alerts and incessant
pointing and clicking.</p>
OSSEC's implementation of FIM is similar to every other product in this
space.  You can monitor changes in modification times, access times,
checksums (MD5 and SHA1), owners, groups, and permissions or any combination
of those attributes.  OSSEC's file integrity events occur inside the same
framework as the log analysis, so it's possible to write rules to evaluate
and correlate them.  It's even possible to fire off scripts when they occur
to perform validation against other databases.</p>
Just this week I discovered this post on Improving file integrity
monitoring with
OSSEC</a>
that allows the servers to keep a validated list of checksums to ignore!
It's also possible to audit the Windows Registry for changes using OSSEC's
file integrity daemon.</p>
Rootkit detection</h3>
Rootkit detection by itself isn't that valuable.  In the context of the
logging events and file integrity data, it can be invaluable</strong> in quickly
identifying compromised servers.  OSSEC uses a rootkit and trojan'd database
of file checksums to detect bad files on your system.  It also performs
checks similar to
chkrootkit</a> or
AIDE</a> by looking for hidden files, programs,
or open ports and checking for out of place files and directories.</p>
Again, the events occur inside the OSSEC framework and can be customized
using the OSSEC rules.</p>
Policy auditing</h3>
If you find yourself in an environment covered by a set of IT regulations,
you may have to perform audits of critical daemons configurations to verify
you are implementing "Industry Best Practices."  While, as a rational human
being, the phrase "industry best practices" makes my skin crawl, security
professionals need to establish and ensure compliance with a security
baseline.  OSSEC provides a policy auditing framework which runs alongside
it's file integrity monitoring daemon to look for both wanted and unwanted
phrases in configuration files.  OSSEC ships with policies for auditing
according to the Security Benchmarks from the Center for Internet
Security</a>.</p>
Alerting and active response</h3>
The real power comes from the flexibility of OSSEC's alerting system.  Using
rules you can modify alert levels and perform aggregate analyis.  It's
simple to write a rule that says "If an IP fails logins 5 times in 10 ten
minutes, alert."  You may recognize this functionality as provided by the
popular Fail2Ban</a> project.  The real power is
OSSEC's analysis can occur across every</em> device on your network.</p>
A detailed overview of the rules syntax can be found on the OSSEC
site</a>.</p>
With the OSSEC active response system, it's possible to implement the
Fail2Ban functionality and blacklist IP's that are misbehaving.  Active
response is a fancy way of saying "when an alert is triggered, run a script
somewhere."  You can specify running the script on the agent that generated
the alarm, a specific agent on your network, or every</em> agent on your
network.</p>
The active response system is incredibly powerful and flexible.  I encourage
you to check the project out, whether or not you buy my book.</p>
Updates</h3>
This post was graciously translated into
Russian</a>
by a volunteer transalation team.</p>


ElasticSearch for Logging
2012-12-26T00:00:00+00:00
We use ElasticSearch</a> at my job for web front-end
searches.  Performance is critical, and for our purposes, the data is mostly
static.  We update the search indexes daily, but have no problems running on
old indexes for weeks.  The majority of the traffic to this cluster is
search; it is a "read heavy" cluster.  We had some performance hiccups at
the beginning, but we worked closely with Shay Bannon of ElasticSearch to
eliminate those problems.  Now our front end clusters are very reliable,
resilient, and fast.</p>
I am now working to implement a centralized logging infrastructure that
meets compliance requirements, but is also useful.  The goal of the logging
infrastructure is to emulate as much of the Splunk functionality as
possible.  My previous write-up on
logging</a>
explains why we decided against Splunk</a>.</p>
After evaluating a number of options, I've decided to utilize ElasticSearch
as the storage back-end for that system.  This type of cluster is very
different</strong> from the cluster we've implemented for heavy search loads.</p>
</span>
Translations</h3>
A Russian translation</a> of this post provided by EveryCloud</a>.

A Chinese translation</a> of this post provided by FangPeishi</a>.

A Ukrainian translation</a> of this post provided by Open Source Initiative</a>.</p>
Index Layouts</h2>
The two popular open source log routing systems are
Graylog2</a> and LogStash</a>.  As of
this writing, the stable</strong> Graylog2 release supports only writing/reading
from a single index.  As I pointed out in a prior article, this presents
enormous</strong> scaling issues for Graylog2.  The 0.10.0 release of Graylog2
will include the ability to index to multiple indexes.  However, my
experience has been with LogStash indexes as that was the only scalable
option in the past.</p>
In order to get the most out of ElasticSearch for logging, you need to use
multiple indexes.  There are a few ways to handle when to rollover the
index, but LogStash's default automatic daily rotation turns out to make the
most sense.  So, you'll have something like:</p>

logstash-2012.12.19</li>
logstash-2012.12.20</li>
logstash-2012.12.21</li>
logstash-THE_WORLD_HAS_ENDED</li>
</ul>
You could keep track of how many documents are in each index.  Then roll after
after a million or billion or whatever arbitrary number you decide, but
you're just creating more work for yourself later.  There are some edge
cases where other indexing schemes maybe more efficient, but for most users,
an index a day is the simplest, most efficient use of your resources.</p>
Get serious, or go home.</h2>
Both LogStash and Graylog2 ship with built-in ElasticSearch implementations.
This is great for demonstration or development purposes.  DO NOT USE THIS
BUILT-IN SERVER FOR REAL PURPOSES!</strong></em>  I am surprised by the number of
LogStash and Graylog2 users ending up in #elasticsearch on irc.freenode.org
who are using the built-in ElasticSearch storage engine and surprised that
it falls over!</p>
Run a standalone ElasticSearch Cluster!</strong></em></p>
You will need separate hardware for this.  Java applications like LogStash
and ElasticSearch are memory and disk-cache intensive.  Commit the hardware
to the log processing boxes and separate</strong> boxes to the ElasticSearch
cluster.  Java has some weird issues with memory.  We've found that you
don't want to go past 32 GB of RAM dedicated to ElasticSearch and reserve
atleast 8 GB to the OS for file-system caching.</p>
My cluster is handling ~60 GB of log data a day in my development
environment with 3 search nodes at 24 GB of RAM each and is underwhelmed.
This brings up the next question, how many servers for my cluster?</em>   Start
with 3 servers in your ElasticSearch cluster.  This gives you the
flexibility to shutdown a server and maintain full use of your cluster.  You
can always add more hardware!</p>
Installing ElasticSearch</h2>
I'm not going to cover installing ElasticSearch, you can read more about
it</a> on
the documentation site.  You may even decided to utilize the .deb or
possibly roll an rpm</a> and
create a recipe for managing ElasticSearch with
Puppet</a> or
Chef</a>.  The only thing I
will say about installation, is despite how much it hurts, it's best to run
ElasticSearch under the Sun
JVM</a>.  This is how the developers
of ElasticSearch run ElasticSearch and so can you!</p>
ElasticSearch Configuration: OS and Java</h2>
There are some things you really</em> need to configure on the host system.
I'm assuming you're running Linux as the host system here.  You should run
ElasticSearch as an unprivileged user.  My cluster runs as the
'elasticsearch' user, so we tweak the kernel limits on processes and memory
in '/etc/security/limits.conf':</p>
# Ensure ElasticSearch can open files and lock memory!
</span>elasticsearch   soft    nofile          65536
</span>elasticsearch   hard    nofile          65536
</span>elasticsearch   -       memlock         unlimited
</span></code></pre>
You should also configure ElasticSearch's minimum and maximum pool of memory
be set to the same value</em>.  This takes care of all the memory allocation at
startup, so you don't have threads waiting to get more memory from the
kernel.  I've built ElasticSearch on a RedHat system and have this in my
'/etc/sysconfig/elasticsearch' which sets environment variables for the
daemon at startup:</p>
# Allocate 14 Gigs of RAM
</span>ES_MIN_MEM=14g
</span>ES_MAX_MEM="$ES_MIN_MEM"
</span></code></pre>
This file is managed by Puppet and sets the memory equal to 50% of the RAM +
2 gigs.  This isn't rocket science, and it's covered in every</strong>
ElasticSearch tuning guide.</p>
ElasticSearch Configuration: elasticsearch.yml</h2>
There are some things we can tune in the 'elasticsearch.yml' file which will
dramatically improve performance for write-heavy nodes.  The first is to set
bootstrap.mlockall</code> to true.  This forces the JVM to allocate all of
ES_MIN_MEM</code> immediately.  This means Java has all the memory it needs at
start up!  Another concern of a write heavy cluster is the imbalance of
memory allocating to the indexing/bulk engine.</p>
ElasticSearch is assuming you're going to be using it mostly for searches,
so the majority of your memory allocation is safe guarded for those
searches.  This isn't the case with this cluster, so by tweaking
indices.memory.index_buffer_size</code> to 50% we can restore the balance we need
for this use case.  In my setup, I also up the refresh interval and the
transaction count for log flushing.  Otherwise, ElasticSearch would be
flushing the translog nearly every second.</p>
The other thing we need to tweak to avoid catastrophic fail is the
threadpool settings.  ElasticSearch will do what it believes is best to
achieve the best performance.  We've found out, in production, that this can
mean spawning thousands upon thousands of threads to handle incoming
requests.  This will knock your whole cluster over quickly under heavy load.
To avoid this, we set the max number of threads per pool; search, index, and
bulk.  The majority of our operations will be bulks, so we give that 60
threads, and other operations 20.  We also set the maximum number of
requests that can queue for processing to 200 for bulk, and 100 for
everything else.  This way, if the cluster becomes overloaded it will turn
down new requests, but it will leave you enough file descriptors and PID's
to ssh into the boxes and figure out what went wrong.</p>
Pulling that all together, here's my config file:</p>
##################################################################
</span># /etc/elasticsearch/elasticsearch.yml
</span>#
</span># Base configuration for a write heavy cluster
</span>#
</span>
</span># Cluster / Node Basics
</span>cluster.name</span>: </span>logng
</span>
</span># Node can have abritrary attributes we can use for routing
</span>node.name</span>: </span>logsearch-01
</span>node.datacenter</span>: </span>amsterdam
</span>
</span># Force all memory to be locked, forcing the JVM to never swap
</span>bootstrap.mlockall</span>: </span>true
</span>
</span>## Threadpool Settings ##
</span>
</span># Search pool
</span>threadpool.search.type</span>: </span>fixed
</span>threadpool.search.size</span>: </span>20
</span>threadpool.search.queue_size</span>: </span>100
</span>
</span># Bulk pool
</span>threadpool.bulk.type</span>: </span>fixed
</span>threadpool.bulk.size</span>: </span>60
</span>threadpool.bulk.queue_size</span>: </span>300
</span>
</span># Index pool
</span>threadpool.index.type</span>: </span>fixed
</span>threadpool.index.size</span>: </span>20
</span>threadpool.index.queue_size</span>: </span>100
</span>
</span># Indices settings
</span>indices.memory.index_buffer_size</span>: </span>30%
</span>indices.memory.min_shard_index_buffer_size</span>: </span>12mb
</span>indices.memory.min_index_buffer_size</span>: </span>96mb
</span>
</span># Cache Sizes
</span>indices.fielddata.cache.size</span>: </span>15%
</span>indices.fielddata.cache.expire</span>: </span>6h
</span>indices.cache.filter.size</span>: </span>15%
</span>indices.cache.filter.expire</span>: </span>6h
</span>
</span># Indexing Settings for Writes
</span>index.refresh_interval</span>: </span>30s
</span>index.translog.flush_threshold_ops</span>: </span>50000
</span>
</span># Minimum nodes alive to constitute an operational cluster
</span>discovery.zen.minimum_master_nodes</span>: </span>2
</span>
</span># Unicast Discovery (disable multicast)
</span>discovery.zen.ping.multicast.enabled</span>: </span>false
</span>discovery.zen.ping.unicast.hosts</span>: [ </span>"logsearch-01"</span>, </span>"logsearch-02"</span>, </span>"logsearch-03" </span>]
</span></code></pre>
ElasticSearch Configuration: Index Templates</h2>
As I stated, I developed this cluster based on LogStash due to the short
comings of the Graylog2 implementation at the time.  This section will
contain the word "logstash", but you can easily adapt this to a Graylog2 or
homemade index mapping.</p>
Since we've decided to create an index a day, there's two ways to configure
the mapping and features of each index.  We can either create the indexes
explicitly with the settings we want, or we can use a template such that any
index created implicitly by writing data to it, has the features and
configurations we want!  Templates make the most sense in this case, you
we'll create them on the now running cluster!</p>
My template settings are:</p>
{
</span>    </span>"template"</span>: </span>"logstash-*"</span>,
</span>    </span>"settings"</span> : {
</span>        </span>"index.number_of_shards" </span>: </span>3</span>,
</span>        </span>"index.number_of_replicas" </span>: </span>1</span>,
</span>        </span>"index.query.default_field" </span>: </span>"@message"</span>,
</span>        </span>"index.routing.allocation.total_shards_per_node" </span>: </span>2</span>,
</span>        </span>"index.auto_expand_replicas"</span>: </span>false
</span>    },
</span>    </span>"mappings"</span>: {
</span>        </span>"_default_"</span>: {
</span>            </span>"_all"</span>: { </span>"enabled"</span>: </span>false </span>},
</span>            </span>"_source"</span>: { </span>"compress"</span>: </span>false </span>},
</span>            </span>"dynamic_templates"</span>: [
</span>                {
</span>                    </span>"fields_template" </span>: {
</span>                        </span>"mapping"</span>: { </span>"type"</span>: </span>"string"</span>, </span>"index"</span>: </span>"not_analyzed" </span>},
</span>                        </span>"path_match"</span>: </span>"@fields.*"
</span>                    }
</span>                },
</span>                {
</span>                    </span>"tags_template" </span>: {
</span>                        </span>"mapping"</span>: { </span>"type"</span>: </span>"string"</span>, </span>"index"</span>: </span>"not_analyzed" </span>},
</span>                        </span>"path_match"</span>: </span>"@tags.*"
</span>                    }
</span>                }
</span>            ],
</span>            </span>"properties" </span>: {
</span>                </span>"@fields"</span>: { </span>"type"</span>: </span>"object"</span>, </span>"dynamic"</span>: </span>true</span>, </span>"path"</span>: </span>"full" </span>},
</span>                </span>"@source" </span>: { </span>"type" </span>: </span>"string"</span>, </span>"index" </span>: </span>"not_analyzed" </span>},
</span>                </span>"@source_host" </span>: { </span>"type" </span>: </span>"string"</span>, </span>"index" </span>: </span>"not_analyzed" </span>},
</span>                </span>"@source_path" </span>: { </span>"type" </span>: </span>"string"</span>, </span>"index" </span>: </span>"not_analyzed" </span>},
</span>                </span>"@timestamp" </span>: { </span>"type" </span>: </span>"date"</span>, </span>"index" </span>: </span>"not_analyzed" </span>},
</span>                </span>"@type" </span>: { </span>"type" </span>: </span>"string"</span>, </span>"index" </span>: </span>"not_analyzed" </span>},
</span>                </span>"@message" </span>: { </span>"type" </span>: </span>"string"</span>, </span>"analyzer" </span>: </span>"whitespace" </span>}
</span>
</span>             }
</span>        }
</span>    }
</span>}
</span></code></pre>
To apply the settings to the cluster, we create or update the template with
a PUT:</p>
curl -XPUT 'http://localhost:9200/_template/template_logstash/' -d @logstash-template.json
</span></code></pre>
Setting to the template to logstash-*</code> means all new indexes created that
start with 'logstash-' will have these settings applied.  I override the
default search behavior by disabling the _all</code> fields search and set the
default attribute to @message</code>.  This field will be the raw syslog message.
It's also the only field that doesn't have the analyzer disabled.  Don't
freak out.  This is saving space and indexing time.  It means searching
other fields in the document will match using exact matches rather than
fuzzy searches, but that's O.K.  We can still get that warm fuzzy feeling by
searching the @message</code> field!  This will dramatically reduce the storage
size.</p>
In previous write-ups, before ElasticSearch 0.19, you may have seen the
"_source": { "compress": true }</code> attribute set.  This is not recommended
for logging data.  This attribute determines whether each</em> document (read:
log message) is stored using compression.  As these documents tend to be
very</em> small, compression doesn't really save much space.  It does cost
extra processing at the time of indexing and retrieval.  It's best to
explicitly disable compression for a logging cluster.  The setting which
enabled store compression in our elasticsearch.yml</code> uses block level
compression which is much more efficient.</p>
Index Settings</h3>
The index settings are tuned to a 3 node cluster.  We can change everything
but the index.number_of_shards</code> on the fly if we need to grow or shrink the
cluster.  This setup isn't exactly perfect, as we sometimes end up with
orphaned (unallocated) shards.  This is easy enough to correct by moving
shards around with the ElasticSearch
API</a>.</p>
Instead of replicating the entire index to the entire cluster, we add
storage capacity as we add nodes.  This way we have a "RAID like" setup for
shard allocation.  I have a 3 node cluster, and I create 3 shards per index.
This means the master or "write" shard can be balanced to one on each node.
For redundancy, I set the number of replicas to one.  This means there are 6
shards for each index.  Each node is only allowed to have 2 shards per
index.</p>
You'll need to experiment with these settings for your needs.  Take into
account how many nodes you can afford to lose before you lose functionality.
You'll need to adjust the number of replicas based on that.  I've gone with
a simple recipe here of simply having 1 shard replica.  This means I can
only spare to have a single node out of the cluster.  So far, I've found
that having number_of_replicas</code> equal to ( 2/3 * number of nodes) - 1 to be
a good number, YMMV.</p>
Automatically Expand Replicas</h3>
It's also best to disable ElasticSearch's default behavior to automatically
expand the number of replicas based on how many nodes are in the cluster.
We assume responsibility for managing this manually and gain performance,
especially when we need to stop or restart a node in the cluster.
Auto-expansion is a great feature for search-heavy indexes with small to
medium data sets.  Without reconfiguring, adding another node will increase
performance.  However, if you have a lot of data in your indexes and this
feature is enabled here's what happens when a node restarts:</p>

Everything is good. Number of replicas = 1.</li>
Node A shuts down</li>
Cluster notices node down, goes yellow

replicas = 0, expected 1</li>
number of nodes now = 1</li>
number of replicas expected = 0 now</li>
</ul>
</li>
Cluster health upgraded to green, Everything Spiffy</li>
Node A comes back online</li>
Cluster sends number of replicas expected and actual for all indexes</li>
Node A realizes it's shards are unnecessary, and deletes data</li>
Cluster increments number of nodes, replicas expected = 1, actual = 0</li>
Node A is  notified that number of replicas is not yet met</li>
Node A replicates every</em> shard back into it's index, over the network</li>
</ul>
As you can see, this is less than desirable, especially with a busy cluster.
Please be aware of this behavior in production and watch your network graphs
when you add/remove nodes from your cluster.  If you see spikes, you may
want to manage this manually.  You lose some of the magic, but you may find
it to be black magic anyways.  By disabling the auto expansion of replicas,
this happens:</p>

Everything is good.</li>
Node A shuts down</li>
Cluster notices node down, cluster status yellow</li>
Cluster health does not recover, expected replicas != actual replicas</li>
Node A comes back up</li>
Cluster sends number of replicas expected and actual for all indexes</li>
Node A notifies cluster that it has copies of shards</li>
Cluster expected and actual replicas now equal, health green</li>
Cluster checksums the shards and replicates any out-of-date shards to Node A</li>
</ul>
This is what most people expect the cluster to do by default, but the logic
involved in determining cluster state makes it difficult to accomplish.
Again, the magic behavior of auto_expand_replicas</code> makes sense in most use
cases, but not in our case.</p>
Maintenance and Monitoring</h2>
I wrote a few scripts</a> for working
with ElasticSearch in a production environment.  It includes exporting
metrics to Graphite and Cacti.  There is also a Nagios monitoring check
which is very configurable.  We use these utilities to keep track of the
performance and health of our various clusters including the logging
cluster.  I'll be updating that in the next few days to include my logstash
index maintenance script.</p>
As you write your data to the log cluster, ElasticSearch is creating Lucence
indexes of the log messages in the background.  There is a buffer of
incoming documents and based on your settings, that data is flushed to a
Lucene index.  Lucene indexes are expensive to create/update, but fast</strong>
to search.  This means a single shard may contain hundreds of Lucence
indexes, often referred to as segments.  These segments can each be searched
quickly, but only one can be processed per thread.  This can begin to have
negative effect on performance.  We have seen a 10% degradation in search
speed with indexes with 20+ segments.</p>
Luckily, ElasticSearch provides an API for optimizing the Lucene
segments</a>.
You shouldn't optimize an index that's currently indexing data.  The new
data will just create more segments on those shards.  So how do we know that
we're done writing to an index?  Well, if you remember, I recommended using
daily indexes.  This means, you can run a cron job daily (or hourly) to
check for any indexes with yesterday's date or older and make sure they're
optimized (or max_num_segments = 1</code>).  If you've chosen some other schema
for creating index names, you've just created more work for yourself.</p>
Future explorations</h2>
This post is substantially longer than I expected.  I'm just scratching the
surface on the design and implementation of ElasticSearch clusters for
logging data.  My cluster will be moving from development into production
soon (thought it currently provides production functionality).  When I do,
I'm going to face some additional challenges and I have a notebook full of
ideas on how to structure indexes and the cluster to handle the load and
some of the privacy-related problems that arise when you suddenly provide
simple, fast access to massive amounts of data.</p>


OSSEC HIDS Extension - Accumulator
2012-11-26T00:00:00+00:00
If you haven't looked at OSSEC HIDS</a>, here's the overview:</p>

OSSEC is a scalable, multi-platform, open source Host-based Intrusion
Detection System (HIDS). It has a powerful correlation and analysis engine,
integrating log analysis, file integrity checking, Windows registry
monitoring, centralized policy enforcement, rootkit detection, real-time
alerting and active response.</p>
It runs on most operating systems, including Linux, OpenBSD, FreeBSD, MacOS,
Solaris and Windows.</p>
</blockquote>
OSSEC is a great product, but I ran into an issue when attempting to fulfill
a require for PCI-DSS which involved reviewing our LDAP logs.  I knew</em>
OSSEC would make this simple.  I started writing a rule and realized I had
hit a significant roadblock.  OpenLDAP logs events as they happen and only
logs data relevant to that particular event.  A connect event has the ports
and IPs, and the bind event contains the username, but only the connection
id is the same in the two events.</p>
</span>
Out of the box, OSSEC can handle multiline events if</strong> those events are a
fixed number of sequential lines (which Windows Event Logging).
Unfortunately, the OpenLDAP logs were not fixed line and the ability to
alert on multiple unsuccessful logins from the same ip is not available for
this type of logging demonstrated below:</p>
A connection is established:</p>
Jan 11 09:26:57 hostname slapd[20872]: conn=999999 fd=64 ACCEPT from IP=10.1.2.37:33957 (IP=10.1.2.2:389)
</span></code></pre>
A bind (or login event):</p>
Jan 11 09:26:57 hostname slapd[20872]: conn=999999 op=0 BIND dn="uid=example,ou=People,dc=example,dc=com" method=128
</span></code></pre>
An unsuccessful login:</p>
Jan 11 09:26:57 hostname slapd[20872]: conn=999999 op=0 RESULT tag=97 err=49 text=
</span></code></pre>
A retry:</p>
Jan 11 09:26:57 hostname slapd[20872]: conn=999999 op=1 BIND dn="uid=example,ou=People,dc=example,dc=com" method=128
</span></code></pre>
Success:</p>
Jan 11 09:26:57 hostname slapd[20872]: conn=999999 op=1 RESULT tag=97 err=0 text=
</span></code></pre>
Connection is closed:</p>
Jan 11 09:26:57 hostname slapd[20872]: conn=999999 op=2 UNBIND
</span>Jan 11 09:26:57 hostname slapd[20872]: conn=999999 fd=64
</span></code></pre>
You can see that all the events have the same conn=999999</code> in the log
event.  My accumulator patch</a> is currently
merged into my OSSEC GitHub Repo</a>
allows events to accumulate data using this connection id by means of a
decoder extension.</p>
The patch replaces the standard openldap</code> decoder with my decoder
extension, which is as simple as adding an <accumulate/></code> tag to every
decoded event you'd like to accumulate data.  Here's the relevant section
from /var/ossec/etc/decoders.xml</code>:</p>
<</span>decoder </span>name</span>=</span>"openldap"</span>>
</span>    <</span>program_name</span>>^slapd</</span>program_name</span>>
</span>    <</span>accumulate</span>/>
</span></</span>decoder</span>>
</span>
</span><</span>decoder </span>name</span>=</span>"openldap-connect"</span>>
</span>    <</span>parent</span>>openldap</</span>parent</span>>
</span>    <</span>prematch</span>>ACCEPT</</span>prematch</span>>
</span>    <</span>regex</span>>^conn=(\d+) fd=\d+ ACCEPT from IP=(\S+):</</span>regex</span>>
</span>    <</span>order</span>>id, srcip</</span>order</span>>
</span>    <</span>accumulate</span>/>
</span></</span>decoder</span>>
</span>
</span><</span>decoder </span>name</span>=</span>"openldap-bind"</span>>
</span>    <</span>parent</span>>openldap</</span>parent</span>>
</span>    <</span>prematch</span>>BIND </</span>prematch</span>>
</span>    <</span>regex</span>>^conn=(\d+) op=\d+ BIND dn="\w+=(\w+),</</span>regex</span>>
</span>    <</span>order</span>>id, dstuser</</span>order</span>>
</span>    <</span>accumulate</span>/>
</span></</span>decoder</span>>
</span>
</span><</span>decoder </span>name</span>=</span>"openldap-result"</span>>
</span>    <</span>accumulate</span>/>
</span>    <</span>parent</span>>openldap</</span>parent</span>>
</span>    <</span>prematch</span>> RESULT </</span>prematch</span>>
</span>    <</span>regex</span>>^conn=(\d+) op=\d+ RESULT </</span>regex</span>>
</span>    <</span>order</span>>id</</span>order</span>>
</span></</span>decoder</span>>
</span></code></pre>
In the above decoder, when the connect event happens, we start an accumulate
cache for the event with a key of "hostname openldap id"</em> which stores srcip.
The subsequently decoded BIND event uses the same key and adds "dstuser" to the
"hostname openldap id"</em> cache.  When we get to a result line, that line pulls
both those values from the accumulator cache and they will be available for the
rules to trigger using a <sameip/></code> aggregation.</p>
Accumulate()</code> requires an id to be extracted by the decoder, and uses that
recurring id to add data to events as they are parsed using an in memory cache.
The default, right now only configurable at build, is to keep the data in
memory for up to 5 minutes.  Old entries are expired from the hash every 100
lookups or 10 minutes, whichever happens first.</p>
Using this patch, as new data comes in, that data is then available to be used
by the rule engine for smarter rules.  Here's an example using the above decoder
configuraiton to alert with 5 unsuccessful login attempts in 60 minutes from the
same IP address:</p>
<</span>rule </span>id</span>=</span>"100000" </span>level</span>=</span>"1"</span>>
</span>    <</span>decoded_as</span>>openldap</</span>decoded_as</span>>
</span>    <</span>match</span>> RESULT tag=97 err=49</</span>match</span>>
</span></</span>rule</span>>
</span>
</span><</span>rule </span>id</span>=</span>"100001" </span>level</span>=</span>"10" </span>frequency</span>=</span>"5" </span>timeframe</span>=</span>"60"</span>>
</span>    <</span>if_matched_sid</span>>100000</</span>if_matched_sid</span>>
</span>    <</span>same_source_ip</span>/>
</span>    <</span>description</span>>Multiple failed-logins from same source IP</</span>description</span>>
</span></</span>rule</span>>
</span></code></pre>
Now, I can close an item on my PCI-DSS backlog!  The other nice piece here is
because OSSEC is distributed, I can do this match across multiple OpenLDAP
servers.  I'm working on an active
response</a> script to
automatically disable accounts across a multi-datacenter, OpenLDAP
infrastructure with proper reporting and automatic Help Desk notificaitons!</p>
I'm looking for testers and any suggestions for improvements on the interface as
there is now a decent amount of interest on the OSSEC development mailing list
for incorporating this feature into the 2.8 release.</p>


Using a ProxyCommand to Leap Frog Your Bastions
2012-10-15T00:00:00+00:00
I do most of my work over SSH.  Even when I'm working in my browser or
pgAdminIII, I'm usually</em> doing that over SSH tunnels.  VPN Software has been
around for quite some time and it's still mostly disappointing and usually run
by the least competent group in any IT department.  I developed a workflow using
SSH from my laptop, either on the corporate network or at home, I can ssh
/directly/ to the server I'm interested in working on.</p>
In order to accomplish this, I have made some compromises.  First off, if I'm
SSH-ing from my home, I am /required/ to type the fully qualified domain names
(FQDN) when workign remotely.  I use the presence of the domain name to activate
the proper leap frogging.  I also decided to use ControlMaster's with SSH that
can leave me with a terminal without a prompt when I forget which shell is my
master.  Overall, the pros outweigh the cons and I'm more productive because of
it.</p>
</span>
ControlMaster</h2>
Using a ControlMaster with ssh allows multiple connections to the same tcp
connection.  This means subsequent connections are much</em> faster to open, but
places a limit on the original connection that all connections riding on it must
be closed before the ControlMaster connection closes.  This may or may not be
desirable, but does come in handy when using ProxyCommand to bounce around
through jump hosts as the connection establishment overhead is removed.</p>
Adding this line to your ~/.ssh/config</code> will enable ControlMaster for all
connections:</p>
# Use Control Master so we type passwords less
</span>ControlMaster auto
</span>ControlPath ~/.ssh/ssh_control_%h_%p_%r
</span></code></pre>
Estasblishing a Jump Host</h2>
I find it's best to alias the jump hosts to host names that don't exist in DNS.
Ideally, I'll never log in to these hosts directly, so I can even forget these
names.  Let's create an alias for our bastion host, lets call it 'bastion' and
it run ssh on 65022.</p>
# Bastion Host
</span>Host bastion
</span>  Hostname corporate-bastion.example.com
</span>  Port 65022
</span></code></pre>
Using Your Aliases</h2>
It's really simple:</p>
# Use the bastion to connect to internal resources
</span>Host *.internal.example.com
</span>    ProxyCommand ssh bastion nc %h %p
</span></code></pre>
If you're configuring your networks in a way that make sense, this configuration
will work from home or work.  Usually, the internal.example.com zones live
inside your DNS search path while you're on the corporate network.  You probably
also have SSH access directly to your servers from the corporate network, so
while at work you:</p>
$ ssh webserver-001
</span></code></pre>
And then once you go home you can access the same server, directly by:</p>
$ ssh webserver-001.internal.example.com
</span></code></pre>
Sure, it's more typing, but it gets you exactly where you want quickly.  Chances
are you are using zsh or bash with autocompletion and you can just hit /tab/
when you get to the first '.' to have the autocomplete work it's magic.</p>
It's Complicated ..</h1>
Sure it is.  It's always more complicated.  And we can achieve the same things
with your insanely complicated series of jump hosts.  Chances are, you've got
some "high security" shit going on with your network, and you need to use jump
hosts internally.  Maybe your external bastion machine only provides SSH access
to the other internal bastions on your network.  Well, that's plenty O.K.</p>
Multiple Jump Hosts</h2>
Again, I like to pick aliases not in DNS to avoid confusion.  DNS should be the
authoritative place for things on your network, so don't collide with it.  I
can't help you if you insist on being stupid.  Let's say we have a setup where
we need to connect to an external bastion, then to our internal bastion host
from the outside.  This gives us multiple layers of security, but can drive a
man insane with all the non-sense required to scp, rsync, or tunnel to those
hosts behind two bastions.</p>
When you have multiple jump hosts in play, ControlMaster comes in handy.  It
dramatically reduces connection time and complexity.  Should one of the bastion
hosts require a two-factor authentication scheme, ControlMaster will make you
life incredibly easy.  Here's an example of how I might set this up:</p>
# Use Control Master so we type passwords less
</span>ControlMaster auto
</span>ControlPath ~/.ssh/ssh_control_%h_%p_%r
</span>
</span># External Bastion Host
</span>Host extbastion
</span>  Hostname external-bastion.example.com
</span>  Port 65022
</span>
</span># Internal Bastion Host
</span>Host intbastion
</span>    ProxyCommand ssh extbastion nc internal-bastion.example.com 22
</span></code></pre>
So everything looks the same as we saw earlier.  So to utilize the two jump
hosts together, we can just chain to the internal bastion host!</p>
# Use the internal bastion from the external bastion to connect to internal resources
</span>Host *.internal.example.com
</span>    ProxyCommand ssh intbastion nc %h %p
</span></code></pre>
And there you go, keep chaining on the jump hosts and it will keep working.
Again, I don't like to overlap my Host aliases with DNS, so make good decisions
while naming your aliases.</p>
Bonus Round!</h2>
As an FYI, you can use environment variables in your ProxyCommands! So, maybe
you do this in your ~/.bashrc</code>:</p>
export </span>SSH_PROXY</span>=</span>'int'
</span>alias </span>extssh</span>=</span>"SSH_PROXY=ext ssh"
</span></code></pre>
And add these entries to your ~/.ssh/config</code>:</p>
# External Bastion Host
</span>Host jumper
</span>  Hostname external-bastion.example.com
</span>  Port 65022
</span>
</span># Internal Bastion Host
</span>Host bastion-int
</span>    Hostname internal-bastion.example.com
</span>
</span># External Access to Internal Bastion Host
</span>Host bastion-ext
</span>    ProxyCommand ssh jumper nc internal-bastion.example.com 22
</span>
</span># Proxy based on environment variable SSH_PROXY
</span>Host *.internal.example.com
</span>    ProxyCommand ssh bastion-$SSH_PROXY nc %h %p
</span></code></pre>
Now you can retrain choose how to use your ssh jump hosts using shell
environment variables:</p>
$ ssh webserver-001.internal.example.com
</span># Desktop -> internal-bastion.example.com -> webserver-001.internal.ample.com
</span>
</span>$ extssh webserver-001.internal.example.com
</span># Desktop -> external-bastion.example.com -> internal-bastion.example.com -> webserver-001.internal.example.com
</span></code></pre>
Enjoy!</p>


Silly Graphite Trick with ElasticSearch
2012-07-09T00:00:00+00:00
First things first.  I've stated that you should drop everything and install
Graphite</a>.  If you didn't already, please do
that now.  Go ahead, I'll wait.</p>
Good?  Good.  I don't frequently insist on anything like I do with Graphite.
There's a lot of reasons for that.  If you don't believe me, please see
@obfuscurity</a>'s awesome Graphite series on
his blog</a>.</p>
When you get back we'll talk about how to monitor
ElasticSearch</a> with Graphite for fun and profit!</p>
</span>
Monitoring ElasticSearch : Baseline</h2>
Today's silly graphite trick involves monitoring for
ElasticSearch</a>.  There are a number of solutions
available for live monitoring of an ElasticSearch cluster,
BigDesk</a> and
Head</a> are great solutions for
running and monitoring your cluster real-time.</p>
However, BigDesk lacks the customization and the incorporation of data from
other sources.  Graphite makes it so brain dead simple to store and track
metrics that it's really only your own fault for not tracking the surface area
of the illuminated portion of the moon.  Then you really</em> can track your
metrics against phases of the moon!  Graphite also allows you to store your
raw data and manipulate it for display purposes.</p>
I strongly recommend you consider
Diamond</a> for polling system metrics
and outputting them to your Graphite infrastructure.  After a lot of fiddling
with the collectd graphite
plugin</a>, we were
forced to go with Diamond because the collectd plugin required a
carbon-aggregator to translate and flatten our metrics namespace into a sane
and useable format.  That worked at first, but running hundreds or thousands
of servers through a carbon-aggregator that has to manipulate almost every
metric doesn't scale.  Even with 20 aggregators running on multiple
collectors, we struggled to map the data real-time.  Diamond allowed us to
scale horizontally by mapping metrics on the end point hosts and shipping them
pickled to a carbon relay.</p>
Getting ES Metrics into Graphite</h2>
So, use Diamond to get base system data into Graphite.  Now, we need to get
some insight into the ElasticSearch daemon itself.  I looked a while back and
was unable to find anything useful, so I made it myself.  ElasticSearch
provides a number of API calls to retrieve statistics, but I chose the Node
Stats</a>
as a starting point for pulling data out of ES.</p>
At the time, we weren't sure that Graphite was going to displace Cacti as our
monitoring solution (well, I was</em>) so my
perf_elastic_search.pl</a>
script will output the metrics in the correct format for Graphite or Cacti.
Though Cacti output is untested because I gave up after 3 or 4 hours of trying
to figure out how to get data into Cacti.</p>
I recommend running this on the ElasticSearch nodes with the --local options
specified on a 1 minute cron job:</p>
   * * * * * /path/to/perf_elastic_search.pl --local --carbon-base=es
</span></code></pre>
My Silly Trick</h2>
Now, if you read through Jason's awesome series of posts, you're probably able
to do a lot of cool stuff with Graphite.  If you're studious, you can do
something far cooler than the parlor trick I want to show now!  Enough
suspense, here it is:</p>
</p>
AWESOME, RIGHT?!@#!@? I know.  So, first off,
drawAsInfinite</a>
is awesome.  How awesome?  Well, I'll tell you.  Any value greater than
zero</strong> is displayed as vertical line on your graph.  Sounds neat right?
Combine that with the
offset</a>
function and you can strike a vertical line anywhere on the graph where a data
point is greater than a certain value.</p>
In the above graph, I've chosen to mark a vertical line anywhere the JVM
paused for greater than 1 second with a vertical line.  BUT WAIT!  ES output
MILLISECONDS, and a lot of times it's pausing for only just a few milliseconds
or even nano seconds.  Enough to be greater than 1, but too low for me to
care.  I really want to know when the JVM pauses for more than 1 second, so I
extract and graph that data like so:</p>
 drawAsInfinite(offset(nonNegativeDerivative(es.searchnode-01.jvm.gc.time_ms), -1000))
</span></code></pre>
</p>
ES reports total time spent in GC as counter, so we use
nonNegativeDerivative</a>
to extract the changes.  Since we're in milliseconds, we substract 1,000 and
now only data points with 1 second or higher will be displayed.  That by
itself is interesting, but where it gets fun is when you start correlating
that data with other events.</p>
I added the red line, which is the size of the JVM Heap:</p>
color(es.searchnode-01.jvm.mem.heap.used_bytes,"red")
</span></code></pre>
</p>
That was interesting, but this box is doing a lot of indexing as it's
receiving my log data, so it can easily hit 7-8,000 messages per second.  So I
wanted to graph time spent indexing along with these points.  The ES tracker
keeps a counter of the time, which increments until restarts which you can see
by drops here:</p>
es.searchnode-01.indices.indexing.time_ms
</span></code></pre>
</p>
No worries, using the
derivative</a>
or
nonNegativeDerivative</a>
functions you extract the change and graph the line.  Since we have restarts,
we're going to use the nonNegativeDerivative function to ignore the drops:</p>
nonNegativeDerivative(es.searchnode-01.indices.indexing.time_ms)
</span></code></pre>
</p>
When we add this back into the graph, you'll notice that the line hugs the X
axis pretty tightly:</p>
</p>
This is mainly because that number is so small compared to the bigger JVM Heap
numbers.  No worries, we'll call
secondYAxis</a>
to cram as much data into these pixels as we can!</p>
secondYAxis(derivative(es.searchnode-01.indices.indexing.time_ms))
</span></code></pre>
</p>
Yay!</p>


Follow-up Central Logging
2012-06-18T00:00:00+00:00
The reaction to my Central
Logging</a>
post has been significantly greater and more positive than I could've
expected, so I wanted to recap some of the conversation that came out of this.
I am pleasantly surprised by most of the comments on the Hacker News
Thread</a>.  So, here's a real quick
recap of the responses I've received.  I will continue this series this
weekend with more technical details.</p>
</span>
So here's some reflections on the feedback so far.</p>
Another Alternative: ELSA</h2>
@spazm</a> recommended checking out
ELSA</a>.  Not alone,
as @_viq</a> echo'd @spazm's suggestion.  This trend
continued on HackerNews with another recommendation for ELSA from
ova</a>.  I figured this warranted an
investigation.  Unfortunately I have not had a chance to play around with ELSA
yet.</p>
I read through the docs and found it's using
Sphinx</a> as the search backend.  From my cursory
research, the main differences between ElasticSearch</a>
and Sphinx seems to be ease in configuration and setup of clusters with
ElasticSearch winning.  That said, Sphinx seems to crush ElasticSearch on single
node search capabilities.  This is based on the limited information I could find
in the few minutes I had to spend on researching it.</p>
I will spend some time researching ELSA as it is a Perl</a>
project.  I am a sucker for Perl apps!</p>
DIY</h2>
A number of folks are currently involved in the evaluation of logging tools.  I
was a bit disheartened by the number of people considering rolling their own.
While I love the idea of reinventing the wheel and have done so many, many
times, I have to agree with Marcus Ranum</a>.
Logging is hard, and you're probably over your head.</p>
I'd urge those going down this road to investigate contributing to Open Source
Software already in this space.  If we could strengthen a few projects in this
sphere rather than just constantly building more disposable wheels, we all win.
Again, believe me, I sincerely understand that you want to build your own,
however, this is more complicated than you can imagine.  Why do I know that?
Because I built my</a>
own</a>
wheels</a> in this space as well!</p>
Splunk vs.???</h2>
I did haphazardly request feedback from someone who's had experience with
Splunk.  I realize now that I really need to write-up and screenshot the
capabilities of the Logstash</a> /
Kibana</a> /
Graphite</a> setup.  I plan on doing that later this
week, so until then I'll assume responsibility for the poor feedback in this
area.  I read the Hacker News comments and got the impression that you either
use Splunk and never use anything else again</em> or</strong> you think Splunk is too
expensive.  Both positions lack the evidence and rigor I was looking to elicit,
but again, my fault.</p>
Thank you!</h1>
A big thank you to everyone who engaged in a discussion or helped spread the
word.  I was surprised at the amazing response.</p>
Translations!</h2>

Vicky Rotarova kindly volunteered to translate this page into Belarussian.  You can find the Belarussian translation here</a>.</li>
Artur Weber</a> kindly volunteered to translate this page into Portuguese. You can find the Portuguese translation here</a></li>
</ul>


Central Logging with Open Source Software
2012-06-17T00:00:00+00:00
I have worn many hats over the past few years: System Administrator,
PostgreSQL</a> and MySQL DBA, Perl</a>
Programmer, PHP Programmer, Network Administrator, and Security
Engineer/Officer.  The common thread is having the data I need available,
searchable</strong>, and visible</strong>.</p>
So what data am I talking about?  Honestly, everything</em>.  System logs,
application logs, events, system performance data, and network traffic data
are key requirements to making any tough infrastructure decision, if not key
to the trivial infrastructure and implementation decisions we have to make
everyday.</p>
I'm in the midst of implementing a comprehensive solution, and this post is a
brain dump and road map for how I went about it, and why.</p>
</span>
Step 1: syslog</h2>
I usually start with a sane solution for transporting the events occurring on
UNIX and Windows servers to a central log host.  You may need to aggregate
events at a data center level depending on throughput.  There are a number of
options available to you for centrally logging with syslog, the favorites seem
to be:</p>

syslog-ng</a></li>
rsyslog</a></li>
</ol>
Both are excellent choices, but if you have a tight budget and are reading
between the lines of popular regulatory policy (SOX,PCI-DSS,FISMA,FERPA,etc),
you may want to give some thought to 2 features in particular: guaranteed
delivery, and encrypted transfer.  These are not hard and fast rules that
auditors check for right now, but they will in the near future.</p>
With rsyslog, both features are available in the open source solution, where
as this is not the case with syslog-ng.  However, rsyslog does not run on
Windows, so if you have a large number of Windows Servers, you probably need
to spend money on a central logging solution anyways.  I am not in this
position, so I choose rsyslog</strong>.</p>
The main drawback to rsyslog is the configuration file syntax.  Syslog-ng
decided to do away with legacy syslog config file syntax in favor of a
readable, sensical format.  Rsyslog, decided to maintain the legacy syslog
configuration syntax and extend it for new features.  This is maddening, but
if you don't have the budget for syslog-ng and need encryption and/or
guaranteed delivery, you can make it work.</p>
Central Log Server</h3>
First, we need to setup a place for our logs to land.  Configuring the rsyslog
central server means configuring where we want the logs to live, and how we'd
like to receive them.  I'm calling this host 'logstorage-01.'</p>
I'll explain the configuration step by step.  The first part sets the default
templates, work directory, and loads the modules we need:</p>
# Rsyslog Defaults
</span>$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
</span>$WorkDirectory /var/run/rsyslog
</span>
</span># Modules
</span>$ModLoad immark
</span>$ModLoad imudp
</span>$ModLoad imtcp
</span>$ModLoad imklog
</span>$ModLoad imuxsock
</span></code></pre>
Now, I establish that I want to listen on tcp and udp 514:</p>
## Enable Listeners
</span>$InputTCPServerRun 514
</span>$UDPServerRun 514
</span></code></pre>
Rsyslog uses templates for both filenames and output.  This is an example of
both.  The RemoteHost template will be used to determine the filename foreach
message that comes in.  The ArcSightFormat is going to be used to reformat the
message in a way that an ArcSight Agent can handle.</p>
# Templates
</span>$template RemoteHost,"/var/log/remote/%HOSTNAME%/%$YEAR%/%$MONTH%-%$DAY%.log"
</span>$template ArcSightFormat,"<%PRI%>%TIMESTAMP% %fromhost-ip% %syslogtag%%msg:::sp-if-no-1st-sp%%msg:::drop-last-lf%\n"
</span></code></pre>
Now, we're ready to start doing things with messages.  The first action I
choose is to discard all connection related messages from snmpd as these
consume a lot of disk space.  You can disable this type of logging in snmpd,
but it also serves a good example of log filtering and the discard action '~'.</p>
# Discard SNMPD Connection Messages
</span>if $programname == 'snmpd' and ( $msg contains 'Connection from UDP' or $msg contains 'Received SNMP packet(s) from UDP' ) then ~
</span></code></pre>
At this point, due to the '~' any message matching snmpd and the strings I've
specified will have been discarded.  I now want to log everything</em> to disk
using my RemoteHost</strong> template.  It is important to note that local syslog
messages will also be caught by this next rule:</p>
# Archival Storage
</span>#    All Messages, locally and remote stored to these rules
</span>*.* ?RemoteHost
</span></code></pre>
The *.* tells rsyslog to log everything, the ?RemoteHost, is the template
used for the file name.  This next rule demonstrates how to send selected
messages to a UDP listener using a message format:</p>
# ArcSight
</span>if $programname == 'named' then @arcsight.example.com;ArcSightFormat
</span></code></pre>
So, in this example, anything from named is forwarded to acrsight.example.com
over udp (@) port 514 (default) using the format (;) ArcSightFormat for the
message.</p>
It's at this point that our log archival and any additional remote forwarding
we need is complete.  The next thing we do is discard any messages not sourced
from 'logstorage-01':</p>
# If not sourced locally, stop processing message.
</span>:source , !isequal , "logstorage-01" ~
</span></code></pre>
So now only local events are left and we implement local logging.  This format
should be familiar to anyone who's worked with syslogd before:</p>
# Local Logging
</span>*.info;mail.none;authpriv.none;cron.none                /var/log/messages
</span>authpriv.*                                              /var/log/secure
</span>mail.*                                                  -/var/log/maillog
</span>kern.*                                                  /var/log/kern.log
</span>cron.*                                                  /var/log/cron
</span>*.emerg                                                 *
</span>uucp,news.crit                                          /var/log/spooler
</span>local7.*                                                /var/log/boot.log
</span></code></pre>
Setting up the clients</h2>
Next, we'd like to receive logs on the central server, so we need to setup our
clients to send messages.  At this point, I'm not configuring encryption of
messages.  I would like guaranteed delivery of the messages to the central log
server.  rsyslog has a few ways to do this, including it's own protocol for
delivery.  I don't need insane amounts of guarantee; using TCP and an on-disk
queue will get me most of the way there and is simple to implement.</p>
So, here's my rsyslog.conf, one step at a time:</p>
# Rsyslog Defaults
</span>$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat
</span>$WorkDirectory /var/run/rsyslog  # Default Location for Work Files
</span>
</span># Modules
</span>$ModLoad immark
</span>$ModLoad imklog
</span>$ModLoad imuxsock
</span></code></pre>
Nothing crazy there, load the modules we need, set the standard template for messages.</p>
# Local Logging
</span>*.info;mail.none;authpriv.none;cron.none                /var/log/messages
</span>authpriv.*                                              /var/log/secure
</span>mail.*                                                  -/var/log/maillog
</span>kern.*                                                  /var/log/kern.log
</span>cron.*                                                  /var/log/cron
</span>*.emerg                                                 *
</span>uucp,news.crit                                          /var/log/spooler
</span>local7.*                                                /var/log/boot.log
</span></code></pre>
Again, nothing earth shattering.  Basic syslogd style capture of messages to
disk.  Now we're ready to send messages to our central log storage server.  So
this would be a good time to remove anything we don't want to send from the
stream.  Again, I've used an snmpd connection message filter as a
demonstration.  Anything matching it will be discarded by the '~'.</p>
# Discard SNMPD Spam
</span>if $programname == 'snmpd' and ( $msg contains 'Connection from UDP' or $msg contains 'Received SNMP packet(s) from UDP' ) then ~
</span></code></pre>
So, now we're ready to actually send log messages to the central server.  The
first thing we need to configure is the on-disk queue.  We do this as follows:</p>
# Remote Logging with On Disk Queuring Enabled
</span>$ActionQueueType LinkedList         # Asynchronous Forwarding Mechanism
</span>$ActionQueueFileName centralwork    # Enable disk mode queue
</span>$ActionResumeRetryCount -1          # Infinite Retries
</span>$ActionQueueSaveOnShutdown on       # Save Queue on Exit for reprocessing
</span></code></pre>
And the last thing we need is a destination for the logs for this queue:</p>
*.*     @@logstorage-01.example.com:514
</span></code></pre>
One thing to point out is the use of '@@', this specifies we want to use TCP.</p>
Reflections</h2>
What we have at this point is a fairly reliable transport of our syslog
messages from our UNIX hosts to our central log server.  Remember, we
configured the central log server with both TCP and UDP listeners.  This means
that for systems which rsyslog doesn't support and may not be able to use TCP
delivery, we can send legacy UDP messages to logstorage-01 and it will work.</p>
To get Windows servers participating, you may want to investigate:
syslog-win32</a>,
S.N.A.R.E.</a>, or
eventlog-to-syslog</a>.  I don't
have much experience with them, but they will communicate to rsyslog in this
setup.</p>
Now that we have rsyslog configured, we could use syslog as the backend for
our application logging.  Don't get too angry, you can always use something
like scribe</a> or
Flume</a> as well.  That's the subject of
another write-up though.</p>
Step 2: Doing something with these messages</h1>
Before going any further, I'd like to address the 1,000 lb Gorilla in the
room, Splunk</a>.  I have never worked with Splunk.
Even for my ~150 servers in my previous job, I exceeded the 500mb of logs
allowed per day.  (I am a fan of syslog, why invent another logging protocol
if there's already one available?)  That being said, people I trust who have
experience with it say it's amazing.</p>
As a matter of fact, I've never encountered someone who's used Splunk and had
anything remotely negative to say about it's performance, scalability, or user
experience.  The only complaint I've ever heard is that it is expensive; not
"organic beef" expensive, but Aston Martin expensive.  If you have that kind
of budget to spend on logging, go for it.  I've been working for far too many
poor companies for too long and could not fathom spending hundreds of
thousands of dollars on logging software.</p>
I am insanely curious about how close to Splunk's interface and utility I can
get with open source software.  What follow is my attempt to do just that.</p>
Graylog2</h2>
When I first started down this road, someone suggested I take a look at
Graylog2</a>.  It's user interface is fantastic and it
leverages cool sounding technologies like "MongoDB" and uses a cartoon gorilla
from The Oatmeal</a>.  When you log in, as the interface
loads it says: "Mounting party hats!"  How cutting edge is that?  Awesome.</p>
I love software that has a sense of humor.  It adds to the user experience.
Under the hood, Graylog2 has a number of awesome features including it's own
log format for passing messages around in a way that allows for easy
serialization and deserialization of data in the log stream.  This format is
GELF</a>.</p>
I didn't have too many problems installing Graylog2, and honestly I was very
happy with the interface and configurability.  I sent only a small stream of
log traffic to it and was able to get data out very quickly.  I also noticed
that the release I downloaded was the first to support
ElasticSearch</a> as a storage backend in addition to
MongoDB.</p>
For those of you unfamiliar with ElasticSearch, it's a clustered full-text
search platform based on Lucene</a>.  If you've
never had the privilege of working with ElasticSearch, I can tell you it is
magic.  I support several ElasticSearch clusters in a production environment
and can tell you first hand it's a wonderful product from my stand point.  The
only complaint I have is it's too much magic.  It makes me feel insignificant
as a system administrator because I have to do very little to support it.
It's also incredibly fast and incredibly scalable.</p>
Well, it's scalable if you design your indexes in a certain way.  And this is
where the show stops for Graylog2.  You see, ElasticSearch uses sharding to
distribute data across the cluster.  You can specify how many shards you want
an index to have when you create it.  You can also specify how many copies of
each shard you'd like to keep across the cluster for redundancy.  You can even
do some neat things like saying "keep a copy at each datacenter and never have
all copies of one shard in the same rack in the same datacenter."  This
means you can scale performance at the time of index creation.  If I have
5 shards, I can scale up to 5 cluster nodes and gain performance, after
that, I'm simply gaining redundancy as only 1 shard will be the master at
any one time.</p>
So why is this a problem with Graylog2?  Well, Graylog2 uses a single index
for it's entire database.  Perhaps this is a side-effect of their relatively
late adoption of ElasticSearch as a backend for log storage.  But it means
that you need to build the index at the time of initial installation to
cope with the load of the logs for the future of your logging solution.
Sound easy?  Well, it's not.  If you have a large volume of logs and you
intend on keeping them around for compliance reasons for a long period of
time, Graylog2's use of ElasticSearch will cause significant performance
problems for you, even if you were to know you need 20 nodes in your
cluster.</p>
So, for a large installation with high volume, I cannot recommend Graylog2.
It's beautiful, it's fun, but the ElasticSearch indexing scheme is currently
broken.</p>
Logstash</h2>
So, what else is there?  Well, there's Logstash</a>.
Logstash is more of a log routing or translation protocol than anything else.
Take a look at the list of inputs, filter, and outputs it supports:</p>
inputs      filters          outputs
</span>-------     -------------    -------------------
</span>amqp        date             amqp
</span>exec        dns              elasticsearch
</span>file        gelfify          elasticsearch_river
</span>gelf        grep             file
</span>redis       grok             ganglia
</span>stdin       grokdiscovery    gelf
</span>stomp       json             graphite
</span>syslog      multiline        internal
</span>tcp         mutate           loggly
</span>twitter     split            mongodb
</span>xmpp                         nagios
</span>zeromq                       null
</span>                             redis
</span>                             statsd
</span>                             stdout
</span>                             stomp
</span>                             tcp
</span>                             websocket
</span>                             xmpp
</span>                             zabbix
</span>                             zeromq
</span></code></pre>
AH HA!  You'll notice that one of the outputs logstash supports is
ElasticSearch.  So why use logstash instead of Graylog2?  It has to do with
the indexes.  Graylog2 implements a single index 'graylog2' in the
ElasticSearch cluster.  This makes the search API fairly simple, as I simply
specify that index to search from and give my filter criteria.  The downside,
this index is ENORMOUS, so simple searches, or unbounded searches could
dramatically impact the availability of the entire cluster.</p>
Logstash's developers seem to have more experience with the ElasticSearch
model and designed it with scalability in mind.  The logstash elasticsearch
output mechanism creates a new index every day</em>.  This means there's a little
more logic needed on the search front-end to specify which indexes to look in
for the data you're querying, but</strong> you can change the sharding definitions
on a daily basis and grow your cluster as your needs change.  This also
allows for index optimization.  Someone much smarter than me can explain
better, but if an index is in a readonly (or infrequent write) state, like
yesterday's index, it can be highly optimized with Lucene to yield better
performance.</p>
Extracting Custom Data</h3>
One of my first uses for Logstash was to provide a better UI for
OSSEC-HIDS</a>.  OSSEC does an amazing job of security
monitoring hosts and aggregations of hosts, but the interface is fairly behind
where I feel it needs to be.  However, I could say the same thing about
Logstash.  That's fine, because we can leverage the strengths of Logstash to
provide better interfaces.</p>
I found this awesome write-up on getting OSSEC alerts to
Logstash</a>
for processing.</p>
Where Logstash fails</h3>
Logstash isn't perfect, it's front-end leaves MUCH to be desired.  However,
the infrastructure and flexibility it affords, I'd prefer the developers focus
on the inputs, filter, and outputs than waste valuable resources on
front-ends.  If you've learned anything in the open source community, someone
will fix that problem.  And it turns out they have:</p>
Searchable and Visible: Kibana</h2>
As I've pointed out, the ElasticSearch storage in logstash is excellent.  It
uses the indexes exactly as they were designed to be used.  This means the
performance, reliability, and scalability of logstash's storage backend is on
par with Splunk.  However, it's front-end has nothing</em> on Splunk.  Enter
Kibana</a>.</p>
Kibana is a Bootstrap</a>-based PHP
front-end which leverages the indexes Logstash creates in ElasticSearch to
provide a beautiful front-end to the log searching.  It also adds in the
functionality to do trending and analysis of logs from Logstash!  Keep in mind
you can create your own fields in logstash using grok, so we can extract,
trend, score, and analyze data in real-time in a fairly beautiful and powerful
interface.</p>
Kibana fills the gap with the Logstash interface so perfectly.  It doesn't
give me everything I'd get with Splunk, but I've just touched the
functionality I can extract with Logstash.</p>
Step 3: All your stats are belong to Graphite</h1>
Graphite</a> is awesome.  If you're not using it,
DROP EVERYTHING AND GET IT RUNNING NOW</strong></em>.  For a foray into it's
awesomeness, here's a quick overview of it's
features</a>.  This
presentation is based off something that Jason
Dixon</a> shared with me, so please make sure
you visit his blog and read his series of misnamed "Unhelpful Graphite Tips":</p>

Unhelpful Graphite Tip #1</a></li>
Unhelpful Graphite Tip #2</a></li>
Unhelpful Graphite Tip #3</a></li>
Unhelpful Graphite Tip #4</a></li>
Unhelpful Graphite Tip #5</a></li>
Unhelpful Graphite Tip #6</a></li>
Unhelpful Graphite Tip #7</a></li>
Unhelpful Graphite Tip #8</a></li>
Unhelpful Graphite Tip #9</a></li>
Unhelpful Graphite Tip #10</a></li>
</ul>
If you've taken the time to read the overview and those tips, you're probably
thinking "OMG I CAN GRAPH EVERYTHING."  If you're not thinking that, re-read
everything and reconsider.  Perhaps you missed
timeShitft()</a>
? Perhaps you're not a wanna-be statistics nerd like me?  I strongly suggest
you become one, maybe start here</a>.</p>
So, you can write grok patterns to extract metrics from your logs, you can
then use statsd</a> to track those metrics in
Graphite.</p>
STFU, Information Overload</h1>
Agreed, this is a lot for a single post, and I still have so much to say on
all the technology here.  I will try to keep brain-dumping as I fine-tune my
setup.  I aimed to answer the question, how close can I get to Splunk with
open source software?  I honestly don't know.  I'm relying on you folks to
find out.  Start a conversation with me on twitter,
@reyjrar</a>.</p>
UPDATE!</h1>

Anja Skrba kindly volunteered to translate this page into Serbian.  You can find his Serbian translation here</a>.</li>
Atilla Debredeni</a> volunteered to translate this page into Hungarian. You can find his Hungarian translation here</a>.</li>
Irakli Nishnianidze</a> volunteered to translate this page into Georgian. You can find his Georgian translation here</a>.</li>
</ul>


Statistics, Risk Analysis, and Misunderstandings
2010-06-11T00:00:00+00:00
I married a Statistician, so this
article</a> sums the
lectures I receive on a daily basis.  Risk Management is statistical analysis,
and I'm not sure how many folks in IT Security have Graduate level Stat
exposure.  So, the understanding of our statistical shortcomings is key.  You
need to read that entire article, twice. </p>
</span>
This statement struck me, as I've noticed a scary trend in IT Security:</p>

"People who know a little bit of statistics - enough to use statistical
techniques, not enough to understand why or how they work - often end up
horribly misusing them.  Statistical tests are complicated mathematical
techniques, and to work, they tend to make numerous assumptions.  The
problem is that if those assumptions are not valid, most statistical tests
do not cleanly fail and produce obviously false results."</p>
</blockquote>
As we outsource more security, and buy more products, we must be careful, as
this statement is also true:</p>

"People who know a little bit of IT Security</em> - enough to use an IDS or
SIEM</em>, not enough to understand why or how they work - often end up
horribly misusing them.  Security tools</em> use complicated technical
techniques</em>, and to work, they tend to make numerous assumptions. The
problem is that if those assumptions are not valid, most security tools do
not cleanly fail and produce obviously false results."</p>
</blockquote>
My wife's constant guidance in Statistics has been invaluable to my evaluations
of IT Security Policy and Implementation.  When I came across this article
thanks to @alexhutton</a>, I had to share it!</p>


Screen Scraping HTML
2005-04-06T00:00:00+00:00
We've all found useful information on the web.  Occassionally, its even
necessary to retrieve that information in an automated fashion.  It could be
just for your own amusement, possibly a new web service that hasn't yet
published an API, or even a critical business partner who only exposes a web
based interface to you.</p>
Of course, screen scraping web pages is not the optimal solution to any
problem, and I highly advise you to look into APIs or formal web services
that will provide a more consistent and intentional programming interface.
Potential problems could arise for a number of reasons.</p>
</span>
Step 0 : Considerations</h2>
Most obvious and annoying problem is you are not guaranteed any form of
consistency in the presentation of your data.  Websites are under
construction constantly.  Even when they look the same, programmers and
designers are behind the scenes tweaking little pieces to optimize,
straighten, or update.  This means that your data is likely to move or
disappear entirely.  As you can imagine, this can lead to erroneous data or
your program failing to complete.</p>
A problem that you might not think of immediately is the impact of your
screen scraping on the target's web server.  During the development phase
especially, you should give serious thought the mirroring the website using
any number of mirroing applications available on the web.  This will protect
against you accidentally Denial of Servicing the target's web site.  Once
you move to production, out of common courtesy, you should limit the running
of your program to as few times as possible to provide you with the accuracy
your required.  Obviously, if this is a business-to-business transaction,
you should keep the other guy in the loop.  It won't be good for your
business relationships should you trip the other companies Intrusion
Detection System and then have to explain what you're to a defensive
security administrator.</p>
Along the same lines, consider the legality of the screen scraping.  To a
web server, your traffic could masquerade as 100% interactive, valid
traffic, but upon closer inspection, a wise system administrator will likely
put the pieces together.  Search that companies website for "Acceptable Use
Policies" and "Terms of Service."  In some cases, they may not apply but it's
likely that the privilege to access the data is granted only after agreeing
to one of the two aforementioned documents.</p>
Step 1 : Research</h2>

At this point, it's necessary to dive into the task at hand.  Go through the
motions manually in a web browser that supports thorough debugging.  My
experience with Firefox</a> has always been
a positive one.  Through the use of tools like the DOM Inspector</a>, the
built in Javascript Debugger, and extensions like Web
Developer</a>, View
Source With ..</a>, and Venkman</a> its been one of
the best platforms for web development I've encountered.  Incidentally, the
elements of web design are critical to the automated extraction of that
data. There are two phases to debug to write a good screen scraper.
</p>
The Request</h3>
A web server is not a mind reader, it has to know what you're after.
HTTP Requests tell the web server what document to serve and how to serve
it.  The request can be issued through the address bar, a form, or a link.
As you navigate the site, take note of the parameters passed in the Query
String of the URL.  If you need to login, use the Web Developer</b>
Extension to "Display Form Details" and take note of the names of the login
prompt and the form objects themselves.  Also, its important to take note of
the "METHOD" the form is going to use, either "GET" or "POST".  As you go
through, sketch out the process on a scrap piece of paper with details on
the parameters along the way.  If you're clicking on links to get where you
need, use the right click option of "View Link Properties" to get details.
</p>

A key thing people often miss when doing web automation is the effect of
client side scripting.  You can use Venkman</b> to step through the
entire run of client side code.  You want to pay attention to hidden form
fields that are often set "onClick" of the submit button, or through other
types of normal user interaction.  Without knowing and setting these hidden
fields to the correct value, the page will refuse to load or cause problems.
Granted, this isn't good practice on the site designer's part as a growing
number of security aware web surfers are limiting, or disabling client side
scripting entirely.
</P>
The Response</h3>
After sketching out the path to your data, you've finally arrived at the
page that contains the data itself.  You now need to map out the page in a
way that your data can be identified from the rest of the insignificant
details, styling, and advertisements!  I've always believed in syntax
highlighting and have become accustomed to vim's</a> flavor of highlighting.  I've got the
View Source With ..</b> Extension configured to use gvim.  So I right
click and with any luck, the page source is displayed in the gvim buffer
with syntax highlighting enabled.  If the page has a weird extension, or no
extension, I might have to "set syntax=html" if its not presenting the
proper page headers.  Search through the source file, correlating the visual
representations in the browser with the source code that's generating them.
You'll need to find landmarks in the HTML to use as a means to guide your
parser through an obscure landscape of markup language.  If you're having
problems, another indispensible tool provided by Firefox</b> is the "View
Selection Source".  To use it, simply highlight some content and then right
click -> "View Selection Source".  A Mozilla Source viewer opens with
just the HTML that generated the selected content highlighted with some
surrounding HTML to provide context.
</p>
You're going to have to start thinking like a machine.  Think Simple, 1's
and 0's, true and false!  I usually start at my data and start working back,
looking for a unique tag or pattern that I can use to locate the data moving
forward.  Look not only at the HTML Elements (<b>,<td>, etc) but
at their attributes (color="#FF000",colspan="3") to profile the areas
containing and surrounding your data.
</p>
The lay of the land is changing these days.  It should be getting much
easier to treat HTML as a data source thanks Web Standards and the alarming
number of web designers pushing whole-heartedly for their adoption.  The old
table based layouts, styled by font tags and animated GIFs is giving way to
"Document Object Model" aware design and styling fueled mostly by Cascading
Style Sheets (CSS).  CSS works most effectively when the document layout
emulates an object.  There are "classes", "ids", and tags establish
relationships.  CSS makes it trivial for Web Designers with passion and
experience in Design Arts, to cooperate with Web Programmers whose passion
is the Art of Programming and whose idea of "progressive design" is white
text on a black background!  The cues that Programmers and Designers specify
to insure interoperability of Content and Presentation gives the Screen
Scraper a legible road map by which to extract their data.  If you see
"div", "span", "tbody", "theader" elements bearing attributes like "class"
and "id" favor using these elements as landmarks.  Though nothing is
guaranteed, it's much more likely that these elements will maintain their
relationships as they're often the result of divisional cooperation than
entropy.
</P>
One of the simplest ways to keep your bearing is to print out the section
of HTML you're targetting, and sketch out some simple logic to be able to
quickly identify it.  I use a highlighter and a red pen to make notes on the
print out that I can glance at as a sanity check.</p>
Step 2 : Automated Retrieval of Your Content</h2>
Depending on how complicated the path to your data, there are a number of
tools available.  Basic "GET" method requests that don't require cookies,
session management, or form tracking can take advantage of the simple
interface provided by the LWP::Simple</a>
package.  </p>
use </span>strict;
</span>use </span>LWP::Simple;
</span>
</span>my </span>$url </span>= </span>q</span>|</span>http://www.weather.com/weather/local/21224</span>|;
</span>
</span>my </span>$content </span>= </span>get $url;
</span>
</span>print </span>$content;
</span></code></pre>
That's it.  Simple.</p>
More complex problems with cookies and login's will require a more
sophisticated tool.  WWW::Mechanize</a>
offers a simple a solution to a complex path to your data with the ability
to store cookies and construct form objects that can intelligently
initialize themselves. An example:
</p>
use </span>strict;
</span>use </span>WWW::Mechanize;
</span>
</span>my </span>$authPage </span>= </span>q</span>|</span>http://www.weather.com</span>|;
</span>my </span>$authForm </span>= </span>'whatwhere'</span>;
</span>my </span>%formVars </span>= </span>(
</span>    </span>where   </span>=> </span>'</span>21224</span>'</span>,
</span>    </span>what    </span>=> </span>'Weather36HourUndeclared'
</span>);
</span>
</span>#
</span># or optionally, set the fields in visible order
</span>my </span>@visible </span>= </span>qw</span>(</span>21224</span>);
</span>
</span>#
</span># Create a "bot"
</span>my </span>$bot </span>= </span>new WWW::Mechanize();
</span>
</span>#
</span># Masquerade as Mac Firefox
</span>$bot->agent_alias(</span>'Mac Mozilla'</span>);
</span>
</span>#
</span># Retrieve the page with our "login form"
</span>$bot->get($authPage);
</span>
</span>#
</span># fill out the form!
</span>$bot->form_name($authForm);
</span>
</span>while</span>( </span>my </span>($k,$v) </span>= </span>each </span>%formVars ) {
</span>    $bot->field($k,$v);
</span>}
</span>#
</span># OR
</span># $bot->set_visible(@visible);
</span>
</span>#
</span># submit the form!
</span>$bot->submit();
</span>
</span>#
</span># Print the Content
</span>print </span>$bot->content();
</span></code></pre>
Step 3 : Data Processing</h2>

There are two main ways to parse markup languages like HTML, XHTML, and XML.
I've always preferred dealing with the "Event Driven" methodology.
Essentially, as the document is parsed, new tags trigger events in the code,
calling functions you've defined with the attributes of the tag included as
arguments.  The content between a start and end tag is handled through
another callback function that you've defined.  This method requires that
you build your own data structures.  The second method parses the entire
document, building a tree like object from it which it then returns to the
programmer as an object.  This second method is very useful when you have to
process an entire document, modify its contents and then transform it back
into markup language.  Usually, a screen scraping program cares very little
for the "entire document" and more for the interesting tidbits, everything
else can be ignored.
</P>
HTML::Parser</h3>
HTML::Parser</a>
is an event driven HTML parser module available on CPAN</a>. Using the above content retrieval code
snippet, delete the "print $bot->content();" line, and insert this code,
with "use" statements at the top for consistency.
</p>
use </span>HTML::Parser;
</span>
</span>#
</span># store the content;
</span>my </span>$content </span>= </span>$bot->content();
</span>
</span>#
</span># variables for use in our parsing sub routines:
</span>my </span>$grabText </span>= </span>undef</span>;
</span>my </span>$textStr </span>= </span>''</span>;
</span>
</span>#
</span># Parser Engine
</span>my </span>$parser </span>= </span>new HTML::Parser(
</span>                </span>start_h </span>=> [ </span>\&</span>tagStart, </span>"tagname, attr" </span>],
</span>                </span>end_h   </span>=> [ </span>\&</span>tagStop, </span>"tagname" </span>],
</span>                </span>text_h  </span>=> [ </span>\&</span>handleText, </span>"dtext" </span>]
</span>);
</span>
</span>#
</span># Call the parser!
</span>$parser->parse($content);
</span>
</span>#
</span># Display the results between the tag
</span>print </span>$textStr;
</span>
</span>#
</span># Handle the start tag
</span>sub </span>tagStart </span>{
</span>        </span>my </span>($tagname,$attr) </span>= </span>@_;
</span>        </span>if</span>((</span>lc </span>$tagname </span>eq </span>'b'</span>) </span>&& </span>$attr->{</span>class</span>} </span>eq </span>'obsTempTextA'</span>) {
</span>                $grabText </span>= </span>1</span>;
</span>        }
</span>}
</span>
</span>#
</span># Handle the end tag
</span>sub </span>tagStop </span>{   $grabText </span>= </span>undef</span>; }
</span>
</span>#
</span># check to see if we're grabbing the text;
</span>sub </span>handleText </span>{
</span>        $textStr </span>.= </span>shift </span>if </span>$grabText;
</span>}
</span></code></pre>
Using this, its simple to extract the temperature from the variable
$textStr.  If you wanted to extract more information, you could use a more
complex data structure to hold all the variables.  The important thing to
remember about the event based model is everything happens linearly.  It's
good practice to keep state, either through a simple scalar, like the
$grabText var above, or in an array or hash.  If you're dealing with data
that's nested in several layers of tags, you might consider something like
this:</p>
my </span>@nestedTags </span>= </span>();
</span>
</span>sub </span>tagStart </span>{
</span>    </span>my </span>($tag,$attr) </span>= </span>@_;
</span>
</span>    </span>if</span>($tag </span>eq </span>$tagWeAreLookingFor) {
</span>        </span>push </span>@nestedTags,$tag;
</span>    }
</span>}
</span>
</span>sub </span>handleText </span>{
</span>    </span>my </span>$text </span>= </span>shift</span>;
</span>
</span>    </span>#
</span>    </span># In here, we can check where in the @nestedTag array we are, and do
</span>    </span># different things based on location
</span>    </span>if</span>(</span>scalar </span>@nestedTags </span>== </span>4</span>) {
</span>        </span>print </span>"Four Tags deep, we found: </span>$text</span>!</span>\n</span>"</span>;
</span>    }
</span>}
</span>
</span>sub </span>tagStop </span>{
</span>    </span>my </span>$tag </span>= </span>shift</span>;
</span>    </span>pop </span>@nestedTags </span>if </span>$tag </span>eq </span>$tagWeAreLookingFor;
</span>}
</span></code></pre>
This model works great for most screen scraping as we're usually
interested in key pieces of data on a page byh page basis.  However,
this can quickly turn your program into a mess of handler subroutines and
complex tracking variables that make managing your screen scraper closer to
voodoo than programming.  Thankfully, HTML::Parser is fully prepared to make
our lives easier by supporting subclassing.</p>
Step 4 : SubClassing for Sanity</h2>
I usually like to have 1 subclassed HTML::Parser class per page.  In that
class I'll include accessors to the relevant data on that page.  That way, I
can just "use" my class where I'm processing the data for that one page and
I can keep the main program relatively clean from unnecessary clutter.
</p>
The following script, uses a simple interface to pull down the current
temperature in Fahrenheit.  The accessor method  allows the user to specify
the units they'd like the temperature back in.
</p>
#!/usr/bin/perl
</span>
</span>use </span>strict;
</span>use </span>LWP::Simple;
</span>
</span>use </span>MyParsers::Weather::Current;
</span>
</span>my </span>$parser </span>= </span>new MyParsers::Weather::Current;
</span>
</span>my </span>$content </span>= </span>get </span>'http://www.weather.com/weather/local/21224'</span>;
</span>
</span>$parser->parse($content);
</span>
</span>print </span>$parser->getTemperature, </span>" degrees fahrenheit.</span>\n</span>"</span>;
</span>print </span>$parser->getTemperature(</span>'celsius'</span>), </span>" degrees celsius.</span>\n</span>"</span>;
</span>print </span>$parser->getTemperature(</span>'kelvin'</span>), </span>" degrees kelvin.</span>\n</span>"</span>;
</span></code></pre>
The script uses a homemade module "MyParsers::Weather::Current" to handle
all the parsing.  The code for that module is provided below.
</p>
package </span>MyParsers::Weather::Current;
</span>
</span>use </span>strict;
</span>use </span>HTML::Parser;
</span>
</span>#
</span># Inherit
</span>our </span>@ISA </span>= </span>qw</span>(</span>HTML::Parser</span>);
</span>
</span>my </span>%ExtraVariables </span>= </span>(
</span>    </span>_found		</span>=> </span>undef</span>,
</span>    </span>_grabText	</span>=> </span>undef</span>,
</span>    </span>temp_F		</span>=> </span>undef</span>,
</span>    </span>temp_C		</span>=> </span>undef
</span>);
</span>
</span>#
</span># Class Functions
</span>sub </span>new </span>{
</span>    </span>#
</span>    </span># Call the Parent Constructor
</span>    </span>my </span>$self </span>= </span>HTML::Parser::new(@_);
</span>    </span>#
</span>    </span># Call our local initialization function
</span>    $self->_init();
</span>    </span>return </span>$self;
</span>}
</span>
</span>#
</span># Internal Init Function to Setup the Parser.
</span>sub </span>_init </span>{
</span>    </span>my </span>$self </span>= </span>shift</span>;
</span>    </span>#
</span>    </span># init() is provided by the parent class
</span>    $self->init(
</span>        </span>start_h	</span>=>  [ </span>\&</span>_handler_tagStart, </span>'self, tagname, attr' </span>],
</span>        </span>end_h	</span>=>  [ </span>\&</span>_handler_tagStop, </span>'self, tagname' </span>],
</span>        </span>text_h	</span>=>  [ </span>\&</span>_handler_text, </span>'self, dtext' </span>],
</span>    );
</span>
</span>    </span>#
</span>    </span># Set up the rest of the object
</span>    </span>foreach </span>my </span>$k (</span>keys </span>%ExtraVariables) {
</span>        $self->{$k} </span>= </span>$ExtraVariables{$k};
</span>    }
</span>}
</span>
</span>#
</span># Accessors
</span>sub </span>getTemperature </span>{
</span>    </span>my </span>($self,$type) </span>= </span>@_;
</span>
</span>    </span>unless</span>( $self->{</span>_found</span>} ) {
</span>        </span>print </span>STDERR </span>"either you forgot to call parse, or the temp data was
</span>    not found!</span>\n</span>"</span>;
</span>        </span>return</span>;
</span>    }
</span>    $type </span>= </span>'fahrenheit' </span>unless </span>length </span>$type;
</span>
</span>    </span>#
</span>    </span># Remove the first character from the temperature string
</span>    </span>my </span>$t </span>= </span>'temp_' </span>. </span>uc substr</span>($type,</span>0</span>,</span>1</span>);
</span>
</span>    </span>return </span>$self->{$t} </span>if </span>exists </span>$self->{$t};
</span>
</span>    </span>print </span>STDERR </span>"Unknown Temperature Type (</span>$type</span>) !</span>\n</span>"</span>;
</span>    </span>return </span>undef</span>;
</span>}
</span>
</span>#
</span># Parsing Functions
</span>sub </span>_handler_tagStart </span>{
</span>    </span>my </span>($self,$tag,$attr) </span>= </span>@_;
</span>    </span>if</span>((</span>lc </span>$tag </span>eq </span>'b'</span>) </span>&& </span>$attr->{</span>class</span>} </span>eq </span>'obsTempTextA'</span>) {
</span>        $self->{</span>_grabText</span>} </span>= </span>1</span>;
</span>        $self->{</span>_found</span>} </span>= </span>1</span>;
</span>    }
</span>}
</span>
</span>sub </span>_handler_tagStop </span>{
</span>    </span>my </span>$self </span>= </span>shift</span>;
</span>    $self->{</span>_grabText</span>} </span>= </span>undef</span>;
</span>}
</span>
</span>sub </span>_handler_text </span>{
</span>    </span>my </span>($self,$text) </span>= </span>@_;
</span>    </span>if</span>($self->{</span>_grabText</span>}) {
</span>        </span>if</span>(</span>my</span>($temp,$forc) </span>= </span>($text </span>=~ </span>/</span>(\d+).*([</span>CF</span>])</span>/)) {
</span>            </span>if</span>($forc </span>eq </span>'C'</span>) {
</span>                $self->{</span>temp_C</span>} </span>= </span>$temp;
</span>                </span>#
</span>                </span># Fahrenheit doesn't really make decimals places useful
</span>                $self->{</span>temp_F</span>} </span>= </span>int</span>((</span>9</span>/</span>5</span>) </span>* </span>($temp</span>+</span>32</span>));
</span>            }
</span>            </span>elsif</span>($forc </span>eq </span>'F'</span>) {
</span>                $self->{</span>temp_F</span>} </span>= </span>$temp;
</span>                </span>#
</span>                </span># Use precision to 2 decimal places
</span>                $self->{</span>temp_C</span>} </span>= </span>sprintf</span>(</span>"</span>%.2f</span>"</span>, (</span>5</span>/</span>9</span>) </span>* </span>($temp</span>-</span>32</span>));
</span>            }
</span>        }
</span>    }
</span>}
</span></code></pre>
Wrapping Up</h2>
HTML can be an incredibly effective transport mechanism for data, even if
the original author hadn't intended it to be that way.  With the advent of
Web Services and Standards Compliant designs utilizing Cascading Style
Sheets, its becoming more and more interoperable and cooperative.  Learning
to use screen scraping techniques can provide a wealth of information for
the programmer to analyze and format to their heart's content.</p>
As an exercise, you might want to expand on the
MyParsers::Weather::Current</code> object to pull additional information from
weather.com's page, and add a few more accessors!  If you'd really like a
challenge, it'd be kind of fun to write a parser for each of the major
weather sites, pull the data for forecasting down, and use a weighted
average based on the individual sites accuracy in the past to get an
"educated guess" at the weather conditions!</p>


Regular Expression Primer
2004-03-24T00:00:00+00:00
"Regular Expression" is a fancy way to say "pattern matcher."  Humans
can match patterns with relative ease.  A machine has a bit more difficulty
deciphering patterns, especially in text.  As computing became more
powerful, the methods for matching text grew into more flexible dialects.</p>
Regular expressions can be one of the toughest concepts to grasp and use
effectively in any programming language.  Perl is no exception as its
regular expressions engine is perhaps the most advanced regex engine in
existence.  Its power and flexibility also serve to confuse and intimidate
many new comers. It is important to understand the Regular Expression engine
as its often the cause of serious bottlenecks in programs of all shapes and
sizes.</p>
</span>
Introduction</h2>
This introduction aims to cover the basics of regular expressions as they
pertain specifically to host and network administration.  There are a large
number of resources available which present regular expressions in a much
broader context.  A few key things to note before proceeding:</p>

"regex" is short for "Regular Expression"</li>
"regex engine" is the component which translates regex into patterns</li>
/abcd/</b> isn't just 'abcd', its 'a' followed by 'b' followed by 'c' followed by 'd'</li>
</ul>
Meta-characters</h2>
Meta-characters are those characters which the regex engine already has
special meaning for.  In order to match these special characters, its
necessary to prefix them with a back slash "".  The following
meta-characters will be covered in depth:</p>


\</th>
Escape the character proceeded</td>
</tr>

.</th>
Match any single</u> character</td>
</tr>

a|z</th>
Matches a</i> or z</i></td>
</tr>

^</th>
Anchor</i>, Matches the beginning of a string</td>
</tr>

$</th>
Anchor</i>, Matches the end of a string</td>
</tr>

*</th>
Quantifier</i>, 0 or more of previous group or
character</td>
</tr>

+</th>
Quantifier</i>, 1 or more of previous group or
character</td>
</tr>

?</th>
Quantifier</i>, 0 or 1 of previous group or
character</td>
</tr>

{a,z}</th>
Quantifier</i>, matches if the previous group was found
between a</i> and z</i> times.</td>
</tr>

{a,}</th>
Quantifier</i>, matches if the previous group was found
atleast a</i> times.</td>
</tr>

{,z}</th>
Quantifier</i>, matches if the previous group was found
no more than z</i> times.</td>
</tr>

{n}</th>
Quantifier</i>, matches if the previous group was found
exactly</u> n</i> times.</td>
</tr>

[abcd]</th>
Character Class</i>, matches if the character in this
position is either an a, b, c, or d</td>
</tr>

[^abcd]</th>
Inverted Character Class</i>, matches if the character in this
position is not</u> an a, b, c, or d</td>
</tr>

(abcd)</th>
Grouping</i>, groups the matches in the parentheses
into a reference</td>
</tr>
</table>
Grouping</h2>
Grouping affords three main benefits, the ability to capture the data
that regex matches, the ability to link several patterns together for
quantifying, and the ability to reference the data matched by that group
later in the regex.  Capturing and linking are the two most common uses for
grouping.  Back references are rarely needed to accomplish most tasks and
are usually presented in a manner beyond the scope of this article.
Matching text is useful, but usually a programmer is searching for a
word, IP address, or URL buried inside of some text relatively positioned
near a distinguishable mark of some sort.  Usually, that mark is not
important to the rest of the program, but the text recovered from knowing
its position is invaluable.  In this case, grouping is used to capture the
important data, while leaving the rest of the regex to be forgotten as soon
as its finished evaluating.  Perl "remembers" the results of the groups in
special variables $1, $2, $3,</b> etc. based on the position of the
opening parenthesis.</p>
my </span>$line </span>= </span>'First Name:     Bob'</span>;
</span>$line </span>=~ </span>/</span>^</span>First Name :</span>\s+(\S+)</span>/;
</span>my </span>$first_name </span>= </span>$1;
</span></code></pre>
$first_name</b> will now contain "Bob"</p>
Using group to link together pieces of a pattern, it's possible to
quantify that group as a whole.  This is the simplest use of grouping and is
incredibly powerful at the same time.</p>
/^</span>(</span>ab</span>)</span>+</span>$/
</span></code></pre>
This regex will match 'ab'</i>, as well as 'abab'</i>, and
'abababababababababababab'</i>.</P>
Character Classes</h2>
Character classes are sets of characters that can be in a set
position.  Assuming a line begins with a number, using a combination of the
"beginning of string" meta-character '^' and a character class which
represents any numeric character, it would be easy to match:</p>
/^</span>[</span>0</span>-</span>9</span>]</span>/
</span></code></pre>
Matches the line.  It may be more desirable to match lines with one
or more numeric characters in the beginning:</p>
/^[0-9]/</code></pre>
If more precision is required, its possible to specify the number
of digits that will satisfy our match using a more specific quantifier:
</p>
/^[0-9]{1,6}/</code></pre>
This will match a line that begins with one to six digits.
Surprisingly, this regex will still match lines with 7,8,9... digits. In
order to match that starts with 1 to 6 digits we need to tell the regex
engine that the next character can't be a digit.</p>
/^[0-9]{1,6}[^0-9]/</code></pre>
Using the Inverse Character Class we can be explicit and avoid any
confusion between our interpretation and the regex's matching.</p>
The caret in the Character Class served to invert the class.  Inside
the Character Class Meta-characters ([]</b>) there are three
meta-characters:</p>


^ 
(as the first character only) Invert the character class</td>
</tr>

\ 
Escape the next character</td>
</tr>

- 
Range modifier, translates a-d to abcd</td>
</tr>
</table>
A simple example:</p>
/[ab]/</code></pre>
Matches a or b</p>
/[^ab]/</code></pre>
Matches any character that's NOT a or b</p>
Breaking down the more complex regex, the engine reads it as:</p>
^</code></pre>
Anchor, tells the engine to start the match at the beginning of the
string</p>
followed by ..</i></p>
[0-9]{1,6}</code></p>
Match 1 to 6 characters that are any of the following:
0,1,2,3,4,5,6,7,8,9</p>
followed by ..</i></p>
^{1</a></sup></code></p>}
any character that is not</u> one of the following:
0,1,2,3,4,5,6,7,8,9</p>
Perl provides aliases to commonly used character classes to save typing
and reduce some of the complexity of regular expression authoring.


Alias</th>
Meaning</th>
Equivalent Character Class</th>
</tr>

\d</th>
Matches a digit</td>
[0-9]</td>
</tr>

\D</th>
Matches a non-digit</td>
[^0-9]</td>
</tr>

\w</th>
Matches a word character, alphanumeric</td>
[a-zA-Z0-9]</td>
</tr>

\W</th>
Matches a non-word character, non-alphanumeric</td>
[^a-zA-Z0-9]</td>
</tr>

\s</th>
Matches a whitespace character</td>
[ \t\r\n\f]</td>
</tr>

\S</th>
Matches a non-whitespace character</td>
[^ \t\r\n\f]</td>
</tr>
</table>
Using these aliases, its possible to rewrite the previous example as:</p>
/^\d{1,6}\D/</code>
Common Character Class Gotcha</h3>
In an attempt to match an IP address, which can contain 4 numbers
ranged from 0 to 255 separated by a period, programmers often try something
along the lines of the following:</p>
/([0-255]\.){3}[0-255]/</code>
At first glance and close examination, its difficult to understand why
this regex does not match what the programmer is attempting to match.  When
using a character class, the key word is character</i>.  In this context,
the regex engine is not concerned with numbers, but characters</i>.  What
this regex optimizes to is:</p>
/([0125]\.){3}[0125]/</code>
This is most assuredly not what was intended.  The range modifier inside
of a character class evaluates the expression "0-255" as "0-2" + "55" or,
"0125" as duplicate entries in a character class are optimized out.  The
regex to properly match an IP address is very complicated and beyond the
scope of this article.  Assuming no one is attempting to enter IP's in the
888.888.888.0/24, a programmer might construct this regex:</p>
/(\d{1,3}\.?){4}/</code>
Stay tuned for an in depth discussion on this regex.</i></p>
Quantifiers</h2>
Quantifiers allow a programmer to specify a determinately or
indeterminately scale the match of instances in their patterns.  There are
four quantifiers:</p>


?</th>
Matches 0 or 1 consecutive instances of the previous group or
character</td>
</tr>

*</th>
Matches 0 or more consecutive instances of the previous group or
character</td>
</tr>

+</th>
Matches 1 or more consecutive instances of the previous group or
character</td>
</tr>

{a,z}</th>
Range quantifier, specify a minimum (a</i>) and a
maximum (z</i>) number of consecutive instances to match</td>
</tr>
</table>
Zero or More (*)</h3>
The '*' quantifier is almost always misused.  Luckily, in most cases
its negligible, but could still have some unexpected results if a programmer
slips.  Given the lines:</p>

    a dog runs</li>
    
the dog jumps</li>
    
aaa is a car club</li>
</ol>
Which lines will successfully match the following regex?</p>
/a*/</code>
Surprisingly, all three lines will match.  The regex engine will
always be able to find "zero or more a's" in any line of text you send
it.</p>
While this may not seem to be incredibly useful, it actually is.
There are times when a programmer needs to match some text if its there, or
just have an empty string or null if that text isn't found.  This is where
the "zero or more" quantifier earns its keep.</p>
One or More (+)</h3>
The '+' quantifier almost always is what a programmer means when they
use the '*' quantifier.  In the previous example:</p>

    a dog runs</li>
    
the dog jumps</li>
    
aaa is a car club</li>
</ol>
Which lines will successfully match the following regex?</p>
/a+/</code>
In this case, only lines 1 and 3 will successfully match as there is
no 'a' one or more times in line 2.  Often times, this is what was intended
when a '*' was used.</p>
Range Modifier ({a,z})</h3>
The range modifier allows the programmer finer grain control of the
number of consecutive matches to consider.</P>

    a dog runs</li>
    
the dog jumps</li>
    
aaa is a car club</li>
</ol>
Which lines will successfully match the following regex?</p>
/a{2,5}/</code>
In this case, only line 3 will successfully match.  The regex engine
is looking for 2 - 5 consecutive a's.</p>
Greedy VS Non-Greedy</h2>
Quantifiers come into two flavors, "Greedy</b>" and
"Non-Greedy</b>".  The only difference between the two is their relative
ambition to match.  Most regex bottle necks are a direct result of poorly
written Greedy</b> or Non-Greedy</b> matches.</p>
The regex engine wants to match every pattern its passed and it will
do everything in its power to match that regex.  This is why regex can slow
down a program so easily.  Misunderstanding the intention of the regex
engine could result in large regex being evaluated millions of times over
formidable text sample.</p>
Greedy Matching</h3>
Greedy</b> matching seems to be the de-facto standard for most
regex tasks.  These quantifiers are dubbed "greedy" because they are very
ambitious and attempt to match as many times as they can while still
allowing the rest of the regular expression to match.  All of the
quantifiers presented thus far are greedy.</p>
my </span>$string</span>=</span>'1 2 3 4 5 6 7 8 9 10 12 12 13 14 15'</span>;
</span>$string </span>=~ </span>/</span>^.*(\d+).*$</span>/;
</span>my </span>$number </span>= </span>$1;
</span></code></pre>
What will $number</b> contain?  Without understanding greedy and
non-greedy, and how the regex engine goes about satisfying the pattern, its
very difficult for a beginner to answer correctly.  The answer is the "5"
highlighted in red below: 


'1 2 3 4 5 6 7 8 9 10 11 12 13 14 15</b></font>'
</code>
This example is confusing to beginners.  Inspecting how the match is
made should clear things up and hopefully take a giant leap towards
understanding regex in general.  Greedy quantifiers match the maximum number
of instances they can on their first pass.  If the rest of the regex fails
as a result of the greedy quantifier, it will give up its bounty, one
character at a time until the entire regex can match.</p>

'^'</b> - Start at the "beginning of string" anchor.</li>
'.*'</b> - Match any character zero or more times.  The regex
engine matches this greedily, until it fails at the end of string.</li>
'(\d+)'</b> - Fails.  There is no string left to match, so the
.*</b> match gives up the character '5' to the regex.</li>
'(\d+)'</b> - Succeeds matching '5' and storing it in $1.</li>
'.*'</b> - Succeeds as it matches any character zero or more</strike>
times.</li>
'$'</b> - Succeeds, anchor position is currently end of string</li>
</ol>
Introducing Non-Greedy Quantifiers</h3>
Non-Greedy quantifiers are the lazy quantifiers.  Where their greedy
counterparts match the maximum number of instances before allowing the regex
engine to continue, non-greedy quantifiers surrender control as soon as the
minimum number of instances is satisfied.  Non-greedy quantifiers will match
more than the minimum only when its necessary to have the entire regex
succeed.  The non-greedy quantifiers are the same as the greedy quantifiers
immediately followed by a '?':</p>


*?</th>
Matches 0 or more consecutive instances of the previous group or
character, non-greedily</b></td>
</tr>

+?</th>
Matches 1 or more consecutive instances of the previous group or
character, non-greedily</b></td>
</tr>

{a,z}?</th>
Range quantifier, specify a minimum (a</i>) and a
maximum (z</i>) number of consecutive instances to match,
non-greedily</b></td>
</tr>
</table>
The previous example can also demonstrate the laziness of the non-greedy
match.</p>
```perl
my $string='1 2 3 4 5 6 7 8 9 10 12 12 13 14 15';
$string =~ /^.*(\d+).*$/;
my $number = $1;
```
What will $number</b> contain this time?  This time the answer is the "1"
highlighted in red below: 


'1</b></font> 2 3 4 5 6 7 8 9 10 11 12 13 14 15'
</code>

'^'</b> - Start at the "beginning of string" anchor.</li>
'.*?'</b> - Match any character zero or more</strike> times.  The regex
engine lazily accepts no characters for this non-greedy match.  It will
allow characters to match this pattern only if those characters must</b>
be matched to satisfy the entire regex.</li>
'(\d+)'</b> - Succeeds matching '1' and storing it in $1.</li>
'.*?'</b> - Succeeds as it matches any character zero or more</strike>
times.</li>
'$'</b> - Fails, position is not currently end of string</li>
'(.*?)'</b> - In an attempt to match the entire regex, .*?</b>
receives the rest of the string one character at a time until the position
is the end of the string.</li>
'$'</b> - Position is now the end of string, the entire regex
matches!</li>
</ol>
Notes on Greed</h2>
There are two dangers when dealing with quantifiers.  Using the
wrong quantifier could match the wrong instance of text by being over or
under zealous in its attempt to position itself to satisfy the entire regex.
It could also lead to huge performance hits as the regex engine back tracks
across itself and the target text several times to find the first</b>
arrangement that satisfies the regex entirely.</p>
The use of ".*</b>" is common with beginner's, and is misused or
entirely unnecessary in most cases.  The use of anchors (^</b>,$</b>)
should allow the programmer enough freedom to maneuver to the regex engine
to the data they are seeking.</p>
Headaches caused by matching the wrong data need to be addressed by
breaking down the regular expression as done in this article.  Remember, the
regex engine wants to match the regular expression at any cost, so long as
its the cheapest route.  The greedy matches will always get the maximum
instances still allowing the regex to match.  The non-greedy matches will
always get the minimum number of instances still allowing the entire regex
to match.</p>
Revisiting the IP Address Match</h2>
Armed with knowledge of the regex engine's inner workings, dissection
of the earlier IP Address match can reveal its short comings:</p>
/(\d{1,3}\.?){4}/</code>
This regex will match an IP Address such as "192.168.0.1".  It will
however, also match strings like "1234567.234".  Anxiously deciding to
optimize the regex to use quantifiers a hapless programmer noted that the
pattern "digit digit digit period" repeated 3 times in an IP Address and
then was followed by a "digit digit digit." The "digit digit digit" was
repeated 4 times in the string!  The "period" happens "0 or 1" times
depending on which octet the cursor is at the end of.  So the attempt to
shorten the regex inadvertently led to it not being as specific as intended.
Had the programmer stopped at "digit digit digit period", they would've had
a workable solution.  Again, this example doesn't account for the fact that
IP addresses max value per octet is 255, but demonstrates a powerful regex.</p>
/(\d{1,3}.){3}\d{1,3}/</code></p>
Literally, "one to three digits followed by a period, three times,
followed by one to three digits."  This regex would be good enough to pick
out IP Address-like strings from text, then validation could be done one
octet at a time.</p>
Closing Notes on Regular Expressions</h2>
Writing the "right" Regular Expression is often very difficult to do
as machines and humans see text patterns completely different.  Humans are
keen to pick up on spatial patterns, while machines are left to process the
text one character at a time.  Learning to read regular expressions exactly
as the engine can help write more efficient, more effective regular
expressions.</p>
In most circumstances it is possible for the programmer to have
access to the data they are attempting to match or extract from using
regular expressions.  A programmer should build their regular expressions
utilizing a relatively complete data set as the template.  Do not attempt to
write a regular expression to solve every problem.  Specialize regular
expressions as much as needed to get them to work right.</p>
The regex engine in Perl is surrounded by millions of useful tools;
Perl.  Do not forget that.  Most regex beginner's are content to solve
everything in a regex.  Questions like "How do I loop in a regex in perl?"
are not as uncommon as one might hope.  Regular Expressions match text, if
looping is necessary, use foreach</b>, for</b>, while</b>, or
until</b>.  Remember Perl is a huge tool chest with a million tools
inside, there's no need to solve everything with a big hammer (or regex)
even if it might be more fun initially.</p>
systemd-resolved is broken

VPNs and Internet Privacy

Step 4: Extend Your Browser</h2>
To enhance your privacy everywhere, regardless of whether you're using a VPN or not, there are some tools freely available.</p>

Translations</h2>

Karolin Lohmus graciously translated this article into Estonian</a>.</li>
Artur Weber & Adelina Domingos graciously translated this article into Portuguese</a>.</li> </ul>

In which he authors a book on OSSEC

Updates</h3> This post was graciously translated into Russian</a> by a volunteer transalation team.</p>

ElasticSearch for Logging

OSSEC HIDS Extension - Accumulator

Using a ProxyCommand to Leap Frog Your Bastions

Silly Graphite Trick with ElasticSearch

Follow-up Central Logging

Translations!</h2>

Vicky Rotarova kindly volunteered to translate this page into Belarussian. You can find the Belarussian translation here</a>.</li>
Artur Weber</a> kindly volunteered to translate this page into Portuguese. You can find the Portuguese translation here</a></li> </ul>

Central Logging with Open Source Software

Statistics, Risk Analysis, and Misunderstandings

Screen Scraping HTML

Regular Expression Primer

divisonbyzero.net

Goodbye, Twitter

My Experience with Burnout

ElasticSearch CLI Tools - Part 1

divisonbyzero.net

Goodbye, Twitter

Deactivating Your Twitter Account</h2> It's sad, I know. But it's time to let go.</p> Here's the most paranoid way to "delete" your account.</p> Change</strong> your Twitter username, it doesn't matter what you change it to.</li> Navigate to "Your account" > "Deactivate account"</strong></li>

My Experience with Burnout

Value your attention</h3> What you focus on is who you become.</p> </blockquote>

ElasticSearch CLI Tools - Part 1

systemd-resolved is broken

VPNs and Internet Privacy

Firefox (All)</h3> Open a new tab</li> Type: about:config</li> Agree you're breaking your warranty</li> In the search bar, type: network.dns.disablePrefetch</li>

Google Chrome (Mac)</h3> Select "Preferences" from the menu bar</li> Select "Advanced"</li>

Step 4: Extend Your Browser</h2> To enhance your privacy everywhere, regardless of whether you're using a VPN or not, there are some tools freely available.</p>

In which he authors a book on OSSEC

ElasticSearch for Logging

OSSEC HIDS Extension - Accumulator

Using a ProxyCommand to Leap Frog Your Bastions

Silly Graphite Trick with ElasticSearch

My Silly Trick</h2> Now, if you read through Jason's awesome series of posts, you're probably able to do a lot of cool stuff with Graphite. If you're studious, you can do something far cooler than the parlor trick I want to show now! Enough suspense, here it is:</p> </p>

Follow-up Central Logging

Central Logging with Open Source Software

Statistics, Risk Analysis, and Misunderstandings

Screen Scraping HTML

Regular Expression Primer

Step 4: Extend Your Browser</h2>
To enhance your privacy everywhere, regardless of whether you're using a VPN or not, there are some tools freely available.</p>
