From: Greg KH < greg < at> kroah.com>
Subject: kernel.org status: hints on how to check your machine for intrusion
Newsgroups: gmane.linux.kernel
Date: 2011-09-30 23:59:24 GMT (1 week, 4 days, 4 hours and 31 minutes ago)
The compromise of kernel.org and related machines has made it clear that
some developers, at least, have had their systems penetrated. As we
seek to secure our infrastructure, it is imperative that nobody falls
victim to the belief that it cannot happen to them. We all need to
check our systems for intrusions. Here are some helpful hints as
proposed by a number of developers on how to check to see if your Linux
machine might be infected with something:
0. One way to be sure that your system is not compromised is to simply
do a clean install; we can all benefit from a new start sometimes.
Before reinstalling any systems, though, consider following the steps
below to learn if your system has been hit or not.
1. Install the chkrootkit package from your distro repository and see if it
reports anything. If your distro doesn't have the chkroot package,
download it from:
http://www.chkrootkit.org/
Another tool is the ossec-rootcheck tool which can be found at:
http://www.ossec.net/main/rootcheck
And another one is the rkhunter program:
http://www.rootkit.nl/projects/rootkit_hunter.html
[Note, this tool has the tendancy to give false-positives on some
Debian boxes, please read /usr/share/doc/rkhunter/README.Debian.gz if
you run this on a Debian machine]
2. Verify that your package signatures match what your package manager thinks
they are.
To do this on a rpm-based system, run the following command:
rpm --verify --all
Please read the rpm man page for information on how to interpret the
output of this command.
To do this on a Debian based system, run the following bash snippet:
dpkg -l \*|while read s n rest; do if [ "$s" == "ii" ]; then echo $n;
fi; done > ~/tmp.txt
for f in `cat ~/tmp.txt`; do debsums -s -a $f; done
If you have a source-based system (Gentoo, LFS, etc.) you presumably
know what you are doing already.
3. Verify that your packages are really signed with the distro's keys.
Here's a bash snippet that can do this on a rpm based system to
verify that the packages are signed with any key, not necessarily
your distro's key. That exercise is left for the reader:
for package in `rpm -qa`; do
sig=`rpm -q --qf '%{SIGPGP:pgpsig}\n' $package`
if [ -z "$sig" ] ; then
# check if there is a GPG key, not a PGP one
sig=`rpm -q --qf '%{SIGGPG:pgpsig}\n' $package`
if [ -z "$sig" ] ; then
echo "$package does not have a signature!!!"
fi
fi
done
Unfortunately there is no known way of verifying this on Debian-based
systems.
4. To replace a package that you find suspect, uninstall it and install
it anew from your distro. For example, if you want to reinstall the
ssh daemon, you would do:
$ /etc/init.d/sshd stop
rpm -e openssh
zypper install openssh # for openSUSE based systems
yum install openssh # for Fedora based systems
Ideally do this from a live cdrom boot, using the 'rpm --root' option
to point rpm at the correct location.
5. From a liveCD environment, look for traces such as:
a. Rogue startup scripts in /etc/rc*.d and equivalent directories.
b. Strange directories in /usr/share that do not belong to a package.
This can be checked on an rpm system with the following bash snippet:
for file in `find /usr/share/`; do
package=`rpm -qf -- ${file} | grep "is not owned"`
if [ -n "$package" ] ; then
echo "weird file ${file}, please check this out"
fi
done
6. Look for mysterious log messages, such as:
a. Unexpected logins in wtmp and /var/log/secure*, quite possibly
from legitimate users from unexpected hosts.
b. Any program trying to touch /dev/mem.
c. References to strange (non-text) ssh version strings in
/var/log/secure*. These do not necessarily indicate *successful*
breakins, but they indicate *attempted* breakins which means your
system or IP address has been targeted.
7. If any of the above steps show possible signs of compromise, you
should investigate further and identify the actual cause. If it
becomes clear that the system has indeed been compromised, you should
certainly reinstall the system from the beginning, and change your
credentials on all machines that this machine would have had access
to, or which you connected to through this machine. You will need
to check your other systems carefully, and you should almost
certainly notify the administrators of other systems to which you
have access.
Finally, please note that these hints are not guaranteed to turn up
signs of a compromised systems. There are a lot of attackers out there;
some of them are rather more sophisticated than others. You should
always be on the alert for any sort of unexpected behavior from the
systems you work with.
thanks,
greg k-h
From: David Miller < davem < at> davemloft.net>
Subject: Re: kernel.org status: hints on how to check your machine for intrusion
Newsgroups: gmane.linux.kernel
Date: 2011-10-01 01:15:20 GMT (1 week, 4 days, 3 hours and 27 minutes ago)
From: Greg KH < greg < at> kroah.com>
Date: Fri, 30 Sep 2011 16:59:24 -0700
> 1. Install the chkrootkit package from your distro repository and see if it
> reports anything. If your distro doesn't have the chkroot package,
> download it from:
> http://www.chkrootkit.org/
>
> Another tool is the ossec-rootcheck tool which can be found at:
> http://www.ossec.net/main/rootcheck
>
> And another one is the rkhunter program:
> http://www.rootkit.nl/projects/rootkit_hunter.html
> [Note, this tool has the tendancy to give false-positives on some
> Debian boxes, please read /usr/share/doc/rkhunter/README.Debian.gz if
> you run this on a Debian machine]
I quickly found that it gives false positives on Fedora15 too.
It thinks one is infected with Suckit.
It's check is essentially "strings /sbin/init | egrep HOME" which
triggers with:
XDG_CONFIG_HOME
XDG_DATA_HOME
HOME=%s
I'm sure chkrootkit might be useful as a guide, but the seeming
pervasiveness of it's false positives make it much less useful than it
could be.
From: Greg KH kroah.com>
Subject: Re: kernel.org status: hints on how to check your machine for intrusion
Newsgroups: gmane.linux.kernel
Date: 2011-10-01 04:54:48 GMT (1 week, 3 days, 23 hours and 49 minutes ago)
On Fri, Sep 30, 2011 at 09:15:20PM -0400, David Miller wrote:
> From: Greg KH kroah.com>
> Date: Fri, 30 Sep 2011 16:59:24 -0700
>
> > 1. Install the chkrootkit package from your distro repository and see if it
> > reports anything. If your distro doesn't have the chkroot package,
> > download it from:
> > http://www.chkrootkit.org/
> >
> > Another tool is the ossec-rootcheck tool which can be found at:
> > http://www.ossec.net/main/rootcheck
> >
> > And another one is the rkhunter program:
> > http://www.rootkit.nl/projects/rootkit_hunter.html
> > [Note, this tool has the tendancy to give false-positives on some
> > Debian boxes, please read /usr/share/doc/rkhunter/README.Debian.gz if
> > you run this on a Debian machine]
>
> I quickly found that it gives false positives on Fedora15 too.
>
> It thinks one is infected with Suckit.
>
> It's check is essentially "strings /sbin/init | egrep HOME" which
> triggers with:
>
> XDG_CONFIG_HOME
> XDG_DATA_HOME
> HOME=%s
>
> I'm sure chkrootkit might be useful as a guide, but the seeming
> pervasiveness of it's false positives make it much less useful than it
> could be.
That's sad, but the combination of the three tools should give some
sense of what is going on if things are really valid or not, hopefully.
It was tough to write this in a way that would be applicable to the
large variety of systems we all use, so little differences like this
will happen. Hopefully people take this as a guideline of things to do
to try to ensure the safety of a system.
thanks,
greg k-h
From: < akwatts < at> ymail.com>
Subject: Re: kernel.org status: hints on how to check your machine for intrusion
Newsgroups: gmane.linux.kernel
Date: 2011-10-01 14:17:51 GMT (1 week, 3 days, 14 hours and 11 minutes ago)
Greg, many thanks for providing these helpful hints for assessing
system integrity.
On Fri, Sep 30, 2011 at 04:59:24PM -0700, Greg KH wrote:
> The compromise of kernel.org and related machines has made it clear that
> some developers, at least, have had their systems penetrated. As we
> seek to secure our infrastructure, it is imperative that nobody falls
> victim to the belief that it cannot happen to them. We all need to
> check our systems for intrusions. Here are some helpful hints as
> proposed by a number of developers on how to check to see if your Linux
> machine might be infected with something:
I understand that git repos are protected from ex-post tampering by a
rolling sha-1 hash. However, is it possible that code submissions were
faked during the intrusion window and pulled by legitimate subsystem
or system managers?
The intrusion on kernel.org has been dated as potentially weeks
before 8/28 which means many tarballs (that common users rely on more
than git) were posted after that.
Can we confirm a few things?
a) do we know have a better estimate on the date of the initial breach?
b) is there any chance that the signing key (517D0F0E) was compromised?
c) can someone with verifiably clean code (i.e. not just downloads from
kernel.org) post checksums (md5,sha1,rmd160) for official tarball
releases since say 3/2011 (both full kernel and patches)?
Many thanks.
~ Andy
From: Greg KH kroah.com>
Subject: Re: kernel.org status: hints on how to check your machine for intrusion
Newsgroups: gmane.linux.kernel
Date: 2011-10-01 14:28:48 GMT (1 week, 3 days, 14 hours and 3 minutes ago)
On Sat, Oct 01, 2011 at 09:17:51AM -0500, akwatts ymail.com wrote:
> Greg, many thanks for providing these helpful hints for assessing
> system integrity.
>
> On Fri, Sep 30, 2011 at 04:59:24PM -0700, Greg KH wrote:
> > The compromise of kernel.org and related machines has made it clear that
> > some developers, at least, have had their systems penetrated. As we
> > seek to secure our infrastructure, it is imperative that nobody falls
> > victim to the belief that it cannot happen to them. We all need to
> > check our systems for intrusions. Here are some helpful hints as
> > proposed by a number of developers on how to check to see if your Linux
> > machine might be infected with something:
>
> I understand that git repos are protected from ex-post tampering by a
> rolling sha-1 hash. However, is it possible that code submissions were
> faked during the intrusion window and pulled by legitimate subsystem
> or system managers?
>
> The intrusion on kernel.org has been dated as potentially weeks
> before 8/28 which means many tarballs (that common users rely on more
> than git) were posted after that.
>
> Can we confirm a few things?
At this time, we are unable to discuss the events that took place due
to an ongoing investigation into the matter. After that is complete, I
will be working to provide a report of what happened, but that will take
some time.
When www.kernel.org and git.kernel.org come back up, the kernels on them
will have been checked to be verified to be correct. Everyone involved
is working as hard as they can to make that happen as soon as is
possible.
> c) can someone with verifiably clean code (i.e. not just downloads from
> kernel.org) post checksums (md5,sha1,rmd160) for official tarball
> releases since say 3/2011 (both full kernel and patches)?
You can do this today yourself from Linus's git tree if you want to,
it's very easy to script. Just watch out for the fact that gzip puts
dates into the header, so you need to check the .tar file, not the
compressed ones.
thanks for your patience,
greg k-h
From: Andrea Arcangeli < aarcange < at> redhat.com>
Subject: Re: kernel.org status: hints on how to check your machine for intrusion
Newsgroups: gmane.linux.kernel
Date: 2011-10-07 09:28:19 GMT (4 days, 19 hours and 5 minutes ago)
On Fri, Sep 30, 2011 at 04:59:24PM -0700, Greg KH wrote:
> If you have a source-based system (Gentoo, LFS, etc.) you presumably
> know what you are doing already.
Gentoo portage updates through mirrors by default are insecure and I'm
not sure everyone knows what's doing already considering it's not the
default and if I talk to people they're not aware about it. So I
thought it's appropriate to send a reminder considering your topic...
To be secure if you use Gentoo you need to add webrsync-gpg to
FEATURES in make.conf and then use only emerge-webrsync (and never use
emerge --sync). Then you should be safe, after that the
SHA1/SHA256/RMD160 of every further download is verified against the
Manifests which have been cryptographically signed. It's very naive
and too insecure to trust any random mirror and emerge --sync should
be abolished and webrsync-gpg should be the default in FEATURES. After
you see "Good signature from" in output from emerge-webrsync you
should be safe. tarsync then speed things up.