Tag: forensics

Prowler 1.6: AWS Security Best Practices Assessment and Forensics Readiness Tool

Toni de la Fuente 1 Comment

It looks like Prowler has become a popular tool for those concerned about AWS security. I just made Prowler to solve an internal requirement we have here in Alfresco. I decided to make it public and I started getting a lot of feedback, pull requests, comments, advices, bugs reported, new ideas and I keep pushing to make it better and more comprehensive following all what cloud security community seems to need.
I know Prowler is not the best tool out there but it does what I wanted it to do: “Take a picture of my AWS account (or accounts) security settings and tell me from where to start working to improve it”. Do the basics, at least. And that’s what it does. I would use other tools to track service change, etc., I discuss that also in my talks.
Currently, Prowler performs 74 checks (for an entire list run `prowler -l`), being 52 of them part of the CIS benchmark.

Digital Forensics readiness capabilities into Prowler 1.6

`prowler -c forensics-ready`
I’m into DFIR, I love it and I read lot about cloud digital forensics and incident response, I enjoy investing my time R&D about that subject. And I’m concerned about random or targeted attacks to cloud infrastructure. For the talk I’m doing today at the SANS Cloud Security Summit 2018 in San Diego, I wanted to show something new and I thought about adding new checks to Prowler related to forensics and how to make sure you have all (or as much) what you need to perform a proper investigation in case of incident, logs that are not enabled by default in any AWS account by the way. Some of those checks are included and well described in the current CIS benchmark for AWS, or even in the CIS benchmark for AWS three tiers web deployments (another hardening guide that is way less popular but pretty interesting too), but there are checks that are not included anywhere. For example, I believe it is good idea to keep record of your API Gateway logs in your production accounts or even your ELB logs, among many others. So when you run  `prowler -c forensics-ready` now you will get the status of your resources across all regions, and you can make sure you are logging all what you may eventually need in case of security incident. Currently these are the checks supported (https://github.com/toniblyx/prowler#forensics-ready-checks):
  • 2.1 Ensure CloudTrail is enabled in all regions (Scored)
  • 2.2 Ensure CloudTrail log file validation is enabled (Scored)
  • 2.3 Ensure the S3 bucket CloudTrail logs to is not publicly accessible (Scored)
  • 2.4 Ensure CloudTrail trails are integrated with CloudWatch Logs (Scored)
  • 2.5 Ensure AWS Config is enabled in all regions (Scored)
  • 2.6 Ensure S3 bucket access logging is enabled on the CloudTrail S3 bucket (Scored)
  • 2.7 Ensure CloudTrail logs are encrypted at rest using KMS CMKs (Scored)
  • 4.3 Ensure VPC Flow Logging is Enabled in all VPCs (Scored)
  • 7.12 Check if Amazon Macie is enabled (Not Scored) (Not part of CIS benchmark)
  • 7.13 Check if GuardDuty is enabled (Not Scored) (Not part of CIS benchmark)
  • 7.14 Check if CloudFront distributions have logging enabled (Not Scored) (Not part of CIS benchmark)
  • 7.15 Check if Elasticsearch Service domains have logging enabled (Not Scored) (Not part of CIS benchmark)
  • 7.17 Check if Elastic Load Balancers have logging enabled (Not Scored) (Not part of CIS benchmark)
  • 7.18 Check if S3 buckets have server access logging enabled (Not Scored) (Not part of CIS benchmark)
  • 7.19 Check if Route53 hosted zones are logging queries to CloudWatch Logs (Not Scored) (Not part of CIS benchmark)
  • 7.20 Check if Lambda functions are being recorded by CloudTrail (Not Scored) (Not part of CIS benchmark)
  • 7.21 Check if Redshift cluster has audit logging enabled (Not Scored) (Not part of CIS benchmark)
  • 7.22 Check if API Gateway has logging enabled (Not Scored) (Not part of CIS benchmark)
Screenshot while running `forensics-ready` group of checks, here only showing 3 of the first checks that are part of that group
I haven’t added yet a RDS logging check and I’m probably missing many others so please feel free to open an issue in Github and let me know!
If you want to check out my slide deck used during my talk at the SANS Cloud Security Summit 2018 in San Diego, look at here: https://github.com/toniblyx/SANSCloudSecuritySummit2018

[ES] Análisis Forense en AWS: introducción

Toni de la Fuente Leave a comment

English version here.

AWS siempre está monitorizando cualquier uso no autorizado de sus/nuestros recursos. Si tienes docenas de servicios ejecutándose en AWS, en algún momento serás avisado de un incidente debido a varias razones como compartir accidentalmente una contraseña en Github, una mala configuración de un servidor que lo hace fácil de atacar, servicios con vulnerabilidades, DoS o DDoS, 0days, etc… Así que debes estar preparado para realizar un análisis forense y/o gestionar la respuesta ante incidentes de tu infraestructura en AWS.

Recuerda, encaso de incidente debes mantener la calma y procura seguir un procedimiento definido, no dejes este proceso en manos del azar porque probablemente tu o tu jefe este lo suficientemente nervioso como para no esperar ni pensar. Siempre es mucho mejor seguir una guía previamente testeada que tu intuición (que ya la usarás después).

ADVERTENCIA: si has llegado a este artículo de forma desesperada haciendo una búsqueda en Google, te recomiendo probar todos los comandos antes en un entorno controlado como tu laboratorio o sistemas de pruebas. Como decía, deberías tener una guia de respuesta a incidentes y proceso de análisis forense antes de que ocurra un incidente.

En este artículo quiero recomendar algunos pasos y trucos que nosotros hemos usado en algún momento. Doy por hecho que tienes el cliente de linea de comandos de AWS ya instalado, si no es así mira aquí: http://docs.aws.amazon.com/general/latest/gr/GetTheTools.html. Todos los comandos están basados en una posible instancia comprometida en EC2 (Linux), pero la mayoría de estos comandos “aws” se pueden usar también para servidores Windows aunque no lo he probado. Todas estas acciones también se pueden realizar mediante la AWS Cosole. Estos serían algunos de los pasos a tener en cuenta:

1) Desactiva o borra el Access Key. Si una AWS Access Key ha sido comprometida  (AWS te lo hará saber en un correo electrónico u tu lo notarás pagando una gran factura) o te das cuenta que lo has publicado en Github:

aws iam list-access-keys
aws iam update-access-key --access-key-id AKIAIOSFODNN7EXAMPLE \
--status Inactive --user-name Bob
aws iam delete-access-key --access-key AKIDPMS9RO4H3FEXAMPLE \
--user-name Bob

2) En caso de que la Key sea comprometida, comprueba si algún recurso ha sido creado usando esa Key, en todas las regiones. Es común ver que alguien ha usado tus claves para lanzar instancias EC2 en otras regiones de AWS así que comprueba todas buscando instancias que te parezcan sospechosas. Aquí un ejemplo para buscar instancias creadas en la región us-east-1 desde el 9 de Marzo de 2016:

aws ec2 describe-instances --region us-east-1 \
--query 'Reservations[].Instances[?LaunchTime>=`2016-03-9`][].{id: InstanceId, type: InstanceType, launched: LaunchTime}'

3) Contacta con el equipo de soporte de AWS y avísales del incidente, están siempre dispuestos a ayudar y en caso necesario escalarán al equipo de seguridad de AWS.

4) Aisla la instancia, en este caso cuando hablo de YOUR.IP.ADDRESS.HERE, puede ser la IP pública de tu oficina o un servidor intermedio donde saltar y hacer el análisis:

    • Crea un security group para aislar la isntancia, ojo con la diferencia entre EC2-Classic y EC2-VPC, apunta el Group-ID
aws ec2 create-security-group --group-name isolation-sg \
--description "Security group to isolate EC2-Classic instances"
aws ec2 create-security-group --group-name isolation-sg \
--description "Security group to isolate a EC2-VPC instance" --vpc-id vpc-1a2b3c4d 
# where vpc-1a2b3c4d is the VPC ID that the instance is member of
    • Configura una regla para permitir SSH solo desde tu IP pública, aunque primero debes saber tu IP pública:
dig +short myip.opendns.com @resolver1.opendns.com
aws ec2 authorize-security-group-ingress --group-name isolation-sg \
--protocol tcp --port 22 --cidr YOUR.IP.ADDRESS.HERE/32
aws ec2 authorize-security-group-ingress --group-id sg-BLOCK-ID \
--protocol tcp --port 22 --cidr YOUR.IP.ADDRESS.HERE/32 
# note the difference between both commands in group-name \
and group-id, sg-BLOCK-ID is the ID of your isolation-sg
    • Los EC2-Classic Security Groups no soportan reglas de trafico saliente (solo entrante). Sin embargo, para EC2-VPC Security Groups, reglas de trafico saliente se puede configurar con estos comandos:
aws ec2 revoke-security-group-egress --group-id sg-BLOCK-ID \
--protocol '-1' --port all --cidr '0.0.0.0/0’ 
# removed rule that allows all outbound traffic
aws ec2 authorize-security-group-egress --group-id sg-BLOCK-ID \
--protocol 'tcp' --port 80 --cidr '0.0.0.0/0’ 
# place a port or IP if you want to enable some other outbound \
 traffic otherwise do not execute this command.
    • Aplica ese Security Group a la instancia comprometida:
aws ec2 modify-instance-attribute --instance-id i-INSTANCE-ID \
--groups sg-BLOCK-ID 
# where sg-BLOCK-ID is the ID of your isolation-sg
aws iam put-user-policy --user-name MyUser --policy-name MyPowerUserRole \
--policy-document file://C:\Temp\MyPolicyFile.json

5) Etiqueta la instancia para marcarla como “en investigación”:

aws ec2 create-tags –-resources i-INSTANCE-ID \
–tags "Key=Environment, Value=Quarantine:REFERENCE-ID"

6) Guarda los metadatatos de la instancia:

    • Más información sobre la instancia comprometida:
aws ec2 describe-instances --instance-ids i-INSTANCE-ID > forensic-metadata.log
or
aws ec2 describe-instances --filters "Name=ip-address,Values=xx.xx.xx.xx"
    • La salida de consola, puede ser útil dependiendo del tipo de ataque o compromiso aunque recuerda que deberías tener un sistema de logs centralizado:
aws ec2 get-console-output --instance-id i-INSTANCE-ID

7) Crea un Snapshot del volumen o volúmenes de la instancia comprometida para el análisis forense:

aws ec2 create-snapshot –-volume-id vol-xxxx –-description "IR-ResponderName- Date-REFERENCE-ID"
    • Ese snapshot no se modificará o montará, sino que trabajaremos con un volumen.

8) Ahora podemos seguir dos caminos, Parar la instancia:

aws ec2 stop-instances --instance-ids i-INSTANCE-ID
    • o Dejarla ejecutándose, si podemos, en cuyo caso deberíamos aislarla también desde dentro  (iptables) y hacer un volcado de la memoria RAM usando LiME.

9) Crea un Volume desde el snapshot:

    • Piensa que región vas a usar, y otras opciones como  –region us-east-1 –availability-zone us-east-1a –volume-type y personaliza los comandos siguientes:
aws ec2 create-volume --snapshot-id snap-abcd1234
    • Toma nota del volumen:
aws ec2 describe-volumes

10) Monta ese volumen en tu distribución favorita de análisis forense y comienza la investigación.

Iré añadiendo más información en futuros artículos, por ahora esto es una introducción adecuada.

Si quieres aprender mucho más sobre este tema, voy a dar un curso online sobre Análisis forense en AWS, GCE y Azure en español con Securizame, más info aquí.

Forense2-1

Algunas referencias que he usado:

https://securosis.com/blog/my-500-cloud-security-screwup

https://securosis.com/blog/cloud-forensics-101

http://www.slideshare.net/AmazonWebServices/sec316-your-architecture-w-security-incident-response-simulations

http://sysforensics.org/2014/10/forensics-in-the-amazon-cloud-ec2/

Forensics in AWS: an introduction

Toni de la Fuente 2 Comments

Spanish version here.

AWS is always monitoring unauthorized usage of their/our resources up in the cloud. If you have dozens of services running on AWS, at some point, you are likely to be warned about a security issue due to a variety of reasons like accidentally sharing a Key in Github, server misconfiguration making it easily exploitable, services with vulnerabilities, DoS or DDoS, 0days, etc… So be ready to perform a forensics and/or incident response to your AWS infrastructure.

 

Remember, in case of a security incident keep calm and follow a predefined procedure, don’t leave the process to a random behavior because probably your boss or you are nervous and unable to wait. It is always much better to have a proven guide to follow that just follow your intuition (you will use your intuition later).

 

WARNING: you may have come to this article in a desperate try looking for a solution in Google, in this case I recommend you to test all commands mentioned here before in your lab environment. You should have an incident response and forensics guide with some information like this before the incident actually happens.

 

In this article I want to write up some recommended steps and also tips and tricks we have “eventually” done. I assume you have AWS command line tools installed correctly otherwise look at here http://docs.aws.amazon.com/general/latest/gr/GetTheTools.html. All commands are based on Linux EC2 compromised server but most of the “aws” CLI commands can also be used for Windows servers (not tested though). If you are wondering if you may perform all actions mentioned below using AWS Console UI, yes, but I think using command line is faster and straightforward to follow in case of an incident:

 

1) Disable or delete the Access Key. If your AWS Access Key has been compromised (AWS will let you know in the communication or in case you noticed that thru a different manner, i.e.: looking at your code published in GitHub)

aws iam list-access-keys
aws iam update-access-key --access-key-id AKIAIOSFODNN7EXAMPLE \
--status Inactive --user-name Bob
aws iam delete-access-key --access-key AKIDPMS9RO4H3FEXAMPLE \
--user-name Bob

2) In case of compromised Key, check if new and unexpected resources have been spin up using the compromised key, in all regions. It is common to see that someone used your compromised Key to launch EC2 instances in any other AWS region, so check all of them looking for new and suspected instances. Here an example to look for new instances launched in us-east-1 since March 9th 2016:

aws ec2 describe-instances --region us-east-1 \
--query 'Reservations[].Instances[?LaunchTime>=`2016-03-9`][].{id: InstanceId, type: InstanceType, launched: LaunchTime}'

3) Contact AWS Support and let them know about the security incident, they are always willing to help and give advice. They also may scale to AWS Security Team if needed.

4) Isolate the forensic instance, in this case when I talk about YOUR.IP.ADDRESS.HERE, it could be your office public IP or an intermediate hosted server to hop off or to do the analysis:

  • Create a security group to isolate your instance, note the difference between EC2-Classic and EC2-VPC, take note of Group-ID
aws ec2 create-security-group --group-name isolation-sg \
--description “Security group to isolate EC2-Classic instances”
aws ec2 create-security-group --group-name isolation-sg \
--description “Security group to isolate a EC2-VPC instance” \
--vpc-id vpc-1a2b3c4d \
# where vpc-1a2b3c4d is the VPC ID that the instance is member of
  • Set a rule to allow SSH access from your public IP only, but first we have to know our public IP:
dig +short myip.opendns.com @resolver1.opendns.com
aws ec2 authorize-security-group-ingress --group-name isolation-sg \
--protocol tcp --port 22 --cidr YOUR.IP.ADDRESS.HERE/32
aws ec2 authorize-security-group-ingress --group-id sg-BLOCK-ID \
--protocol tcp --port 22 --cidr YOUR.IP.ADDRESS.HERE/32 \
# note the difference between both commands in group-name \
and group-id, sg-BLOCK-ID is the ID of your isolation-sg
  • In EC2-Classic Security Groups don’t support outbound rules. However, for EC2-VPC Security Groups, outbound rules can be set with these commands:
aws ec2 revoke-security-group-egress --group-id sg-BLOCK-ID \
--protocol ‘-1’ --port all --cidr ‘0.0.0.0/0’ \
# removed rule that allows all outbound traffic
aws ec2 authorize-security-group-egress --group-id sg-BLOCK-ID \
--protocol ‘tcp’ --port 80 --cidr ‘0.0.0.0/0’ \
# place a port or IP if you want to enable some other \
outbound traffic otherwise do not execute this command.
  • Apply that Security Group to the compromised instance:
aws ec2 modify-instance-attribute --instance-id i-INSTANCE-ID \
--groups sg-BLOCK-ID \
# where sg-BLOCK-ID is the ID of your isolation-sg
aws iam put-user-policy --user-name MyUser --policy-name MyPowerUserRole \
--policy-document file://C:\Temp\MyPolicyFile.json

5) Tag instance to mark it as under investigation:

aws ec2 create-tags --resources i-INSTANCE-ID \
--tags “Key=Environment, Value=Quarantine:REFERENCE-ID”

6) Save instance/s metadata:

  • Information about the compromised instance:
aws ec2 describe-instances --instance-ids i-INSTANCE-ID > forensic-metadata.log
or
aws ec2 describe-instances --filters “Name=ip-address,Values=xx.xx.xx.xx”
  • Console output, can be useful depending on the attack but you should have a centralized/dedicated log server outside each instance.
aws ec2 get-console-output --instance-id i-INSTANCE-ID

7) Create Snapshot of the volume/s on the compromised instance/s for forensics analysis:

aws ec2 create-snapshot –-volume-id vol-xxxx \
–-description “IR-ResponderName- Date-REFERENCE-ID”

That snapshot won’t be changed or mounted, we will work with a Volume.

8) Now we can follow 2 paths: Stop the instance.

aws ec2 stop-instances --instance-ids i-INSTANCE-ID
  • or Leave it running, if we can, then isolate it from inside (iptables) and dump its RAM memory to a file using LiME.

9) Create a Volume from the taken snapshot to be used later for analysis:

  • Consider using –region us-east-1 –availability-zone us-east-1a –volume-type standard with your own setup.
aws ec2 create-volume --snapshot-id snap-abcd1234
  • Now take note of your new volume:
aws ec2 describe-volumes

10) Mount that volume with your favorite forensics distribution and Run the investigation.

I will add more information in next blog post but I think this is a good introduction.

If you want to learn much more on this topic, I will be giving an online training about AWS, GCE and Azure Forensics in Spanish with Securizame, more info here.

Some cool references and good reads:

https://securosis.com/blog/my-500-cloud-security-screwup

https://securosis.com/blog/cloud-forensics-101

http://www.slideshare.net/AmazonWebServices/sec316-your-architecture-w-security-incident-response-simulations

http://sysforensics.org/2014/10/forensics-in-the-amazon-cloud-ec2/