Amazon AWS

Aus Peter Fuerholz' Wiki
Zur Navigation springen Zur Suche springen

General

Abbrevations / Glossary used by AWS

Beanstalk
cloud services for hosting Java apps
S3
file-system based storage
EBS
Elastic Block Store
EC2
Elastic Compute Cloud
VPC
Virtual private cloud
CloudWatch
Tools for monitoring your apps
Elastic MapReduce
CloudFront
CloudFormation
RDS
Relational Database Service; allows to run an Oracle- or MySQL database.
ElastiCache
in-memory cache web service
SQS
Simple Queue Service
IAM
Identity and Access Management
SNS
Simple Notification Service
SES
Simple Email Service
Route 53
Configure your specific DNS settings.
DynamoDB
non-relational database service
Storage Gateway
Backup tool (?)
SWF
Welcome to Amazon Simple Workflow Service

Developer Tools

There are several developer tools which are needed or useful. Go here to download and install following tools:

IAM Command Line Toolkit: See IAM-Tools

You will need this tool for uploading SSL certificates. Installation:

  1. Download zip from IAM and unzip it to ~/AWS.
  2. Copy ~/AWS/IAMCli-1.4.0/aws-credential.template to aws-credentials-pfu and set your access and secret key accordingly. The access and secret key can be found in the AWS Web Console under the top right menu (press the triangle left to 'Help'), 'Security Credentials'. (Press 'Show' for the secret key.)
  3. sudo gedit /etc/environment and add:
    # for AWS (Amazon Web Services):
    export AWS_IAM_HOME=/home/pfu/AWS/IAMCli-1.4.0
    export AWS_CREDENTIAL_FILE=$AWS_IAM_HOME/aws-credentials-pfu
    Add at file end: PATH=$PATH:$AWS_IAM_HOME/bin
  4. If you access the Internet over a proxy server configure file client-config.template accordingly and add environment variable CLEINT_CONFIG_FILE (see /home/pfu/AWS/IAMCli-1.4.0/README.txt).

Map an Amazon web application as subdomain

There are at least to ways how to accomplish this:

  1. Use Elastic IP: Stackoverflow and Amazon. Do mind that Elastic IP is not free.
  2. Use Route 53: Documentation / Overview. See p. 32 'Migrating a Subdomain to Route 53 without Migrating the Parent Domain'

I chose to use 'Route 53'.

Map web application by means of Route 53

In the AWS Management Console:

  1. Go to tab 'Route 53'
  2. 'Create Hosted Zone', Domain Name: fw-tools.neshendra.ch, Comment: <doesn't matter>
  3. Double-click new added zone table entry: You should now see 2 record, one NS and one SOA
  4. 'Create Record Set' and add an alias to your application: Type: 'A', Alias: 'Yes', Alias target: <choose one of the proposed nameservers> (Do appear only if application is in same AWS context.), Alias Hosted Zone ID: value should appear automatically.

At your domain name provider (at the example of Hostpoint):

  1. Login to Hostpoint control panel, go to 'Domains'
  2. At 'neshendra.ch' select 'Nameserver anpassen' and add 4 NS entries according to the name servers shown in the Amazon Managment console
  3. At Hostpoint the nameserver entries are supervised by an administrator and you get an email if has been proceeded.
  4. At the end you should have following 4 entries:
    fw-tools.neshendra.ch NS 0 3600 <ns1>.awsdns-...co.uk
    fw-tools.neshendra.ch NS 0 3600 <ns2>.awsdns-...com
    fw-tools.neshendra.ch NS 0 3600 <ns3>.awsdns-...net
    fw-tools.neshendra.ch NS 0 3600 <ns4>.awsdns-...org
    (Amazon recommends a timeout of 900 instead of 3600.)

Secure your application by means of TLS/SSL

See port settings and install certificate. See chapter below how I created a certificate.
Off topic: Securing WordPress authenticated area.

Create and add a certificate

Prerequisite: Make sure to have 'IAM Command Line Toolkit' installed (see above).
See Install certificates in AWS
See Configure Elastic Load Balancing

See Overview linked to by Amazon. CACerts would provide free certificates, but the show stopper is, that most browsers do not have this root certificate included (see Wikipedia). See here for certificate authorities included in Firefox. Thus, I opted for StartSSL. (Other good (cheaper) choices might be Simple Authority, GoDaddy and RapidSSL.) With StartSSL you can start for free but: If you want to revoke a certificate you have to pay US$ 24.90 or go for the class 2 certificate which costs you $60.-- for a 2-years certificate.

If you want to create a certificate at StartSSL go on like this:

  1. Go to StartSSL, control panel and register yourself. (Proceed as described.)
  2. Backup the certificate added to your browser: 'Edit', 'Preferences', 'Advanced', 'Encription', 'View certificates', select the certificates under "Startcom", press 'Export'
  3. Go to Startcom control panel and validate your URLs: Select tab 'Validations Wizard' validate all necessary URLs.
  4. Create a private key and Certificate Signing Request (CSR) on the command line (StartSSL does not allow to create a certificate without a password but AWS requires this.) Therefore you need to have OpenSSL installed.
    openssl genrsa 2048 > private-key.pem
    openssl req -new -key private-key.pem -out csr.pem
  5. Go back to Startcom control panel and switch to tab 'Certificates Wizard' and execute wizard for server certificate.
  6. Skip CSR creation and paste the full containment of file 'csr.pem' into the assigned textarea.
  7. Select your 'root domain' and then enter your subdomain to be secured.
  8. Download the certificate and the other referenced documents.
  9. Register the certificate in AWS with following command:
    iam-servercertupload -d -v -b ssl.crt -c sub.class2.server.ca.pem -k private-key.pem -s <subdomain>.<domain> (example for -s: fw-tools.neshendra.ch)
    -d = debug, -v = verbose
  10. You can recheck if everything is OK with:
    iam-servercertgetattributes -s <subdomain>.<domain>
  11. Notice the arn:aws:iam::..... (ends with domain name) identification, then go to AWS Management Console, Elastic Beanstalk, Actions, 'Load/Edit configuration', 'Load Balancer' and put the arn-number (without the number after the domain name) in the field 'SSL Certificate ID'. Set HTTPS Listener Port-field to '443', then Save. (See AWS for further details.)

Now the domain should be accessible with https without any warning from your browser.

Renew your certificate

(Example when using StartCom certificate.)

  • StartSSL will send you an email when the certificate is about to expire.
  • Renew your certificate short before it expires since the new and old timely overlap. If you are too late you are unable to login into www.startssl and you have to register again (and remove your invalid certificate from your browser immediately).

Proceed like this:

  1. Go to StartSSL and press 'Authenticate'. If this does not work (certificate is expired.?) re-register yourself with last email used (peter.fuerholz@neshendra.ch).
  2. Proceed as described (select 'High Grade'). (Appears when login succeed only.)
  3. Fetch installed (public) certificate by backing it up from your (Firefox-)browser: Edit, Advanced, Encryption, View Certificates, select new certificate, backup to pfu/Documents/neshendra/Certificates/SSL-Startcom/<year>/certificate; enter certificate backup password: <see KeePassX>
  4. Copy shell scripts from last certification creation (e.g. pfu/Documents/neshendra/Certificates/SSL-Startcom/<year-1>)
  5. Execute 1createPrivateKeyAndCsr This creates csr.pem (and private-key.pem).
  6. Validate URLs: In Startcom select 'Validations Wizard', 'Domain Name Validation', enter 'neshendra.ch', 'postmaster@...', enter validation code (received by email). (Maybe this step is not necessary every year.)
    If you cannot 'Authenticate' on the Startcom homepage your browser tries to authenticate with your outdated certificate (although you may already have a newer, correct one). In this case delete your old certificate (see next chapter).
  7. Switch to tab 'Certificates Wizard' and execute wizard for server certificate (Web Server SSL/TLS Certificate).
  8. Skip CSR creation step
  9. Enter file content of file csr.pem (created in step 5)
  10. select 'neshendra.ch' (just validated before), then enter 'fw-tools'
  11. Save certificate as ssl.crt. Save intermediate certificate as sub.class1.server.ca.pem and root as ca.pem
  12. In my case sub.class1.server.ca.pem was invalid (not ASCII) and I used the intermediate from 2 years before. (All StartCom-certificates can be downloaded [www.startssl.com/certs/ here] as well.)
  13. Edit the certificate name in script '2uploadCertificate': -s fw-tools<year>.neshen.... (Certificate is not allowed to be overwritten.)
  14. Execute 2uploadCertificate
  15. Edit 3verifyCertificate and execute it as well.
  16. Go to AWS Management Console (https://aws.amazon.com), Elastic Beanstalk, 'Lodur AdF-Dienstleistungsexport', 'neshendra fwtools', 'Configuration' (at the left), 'Load Balancer' and select the new certificate name (e.g. fw-tools<year>.neshendra.ch) in the field 'SSL Certificate ID', 'Save'.
  17. Check: Open 'https://fw-tools.neshendra.ch', click on padlock (left to URL), 'More information...', 'View Certificate' and check that date period is updated.

(AWS guide to update certificate can be found here.)

Remove old certificate in Firefox browser

Normally certificates can be removed in the menu like this:

  • 'Edit', 'Preferences'
  • 'Advanced', 'Certificates', 'Show Certificates'
  • select Certificate to remove, 'Delete or Distrust...' (maybe 'Export' as backup before)

Otherwise:

  • Make sure having package 'libnss3-tools' installed (via Package Manager)
  • Command line options of certutil.
  • Command line:
 cd ~/.mozilla/firefox/kuaimwpt.default
 cp cert8.db cert8.db.backup  # make backup
 certutil -L -d .             # shows all certificates, copy name to be removed afterwards
 certutil -D -n "<copied name>" -d .
  • Restart Firefox

SSH your EC2 instances

It is possible to set up an SSH connection to the server instances (EC2) in Beanstalk. See Elastic Beanstalk doc p. 20f for further details

Open AWS Management Console and go on like this: If not already done:

  1. Go to tab 'EC2', 'Key Pairs' and create a key pair. Set a name and save the private key file (e.g. under neshendra.pem). Reduce the file permissions to 400 (otherwise AWS is complaining on SSH). (This is best done by Midnight Commander.)
  2. Go to tab 'Beanstalk', 'Actions', 'Edit/Load Configurations' and enter the set name of the key pair in the field 'Existing Key Pair', e.g. "neshendra"
  3. Now you have to wait until the server instances have been restarted (takes a minute or two). (Check in EC2-tab.)
    End of 'If not already done
  4. Go to tab 'EC2', 'Instances', right-mouse-click and select 'Connect'. Copy the shown command line and enter this in the shell in the same place where the private key pair is placed to. (This is in my case ~/Documents/neshendra/Certificates/SSL-StartCom/AWS-private-key.)
  5. Now you should be connected to one of your server instances.

See following Q&A.

Inspect Log Files

Log files are located under (see AWS help):

  • Tomcat 6: /var/log/tomcat6/tail_catalina.log
  • Tomcat 7: /opt/tomcat7/logs

If you switched on log rotation by means of an S3 bucket your files are located on S3 in the same bucket as your war file lies (in my case): neshendra-fwtools/logs/i116.../

In my case go on like this:

  1. Open AWS management console, go to tab 'EC2', 'Instances', select instance and press 'Connect'-button above the server overview. Copy ssh command string
  2. open shell and go to your pem file (cd Documents/neshendra/Certificates/SSL-StartCom/AWS-private-key/)
  3. paste ssh command string, Enter
  4. cd /var/log/tomcat6/
  5. less tail_catalina.log
  6. exit

If you enabled copying log files to S3 (under Beanstalk) this file could be found there as well. The file content is logged with UTC (universal time). (During summer time 2 hours behind MEZ.)

File Transfer

For file transfer you have following variants:

  • Without .pem file (less secure): See Create additional user
  • With Nautilus. Since my Linux is rather old my Nautilus (2.30.1) did not support to configure the pem file.
  • By means of Filezilla:
    • Install (via Package Manager) filezilla and putty-tools.
    • Convert .pem by means of the putty-tool: puttygen neshendra.pem -o neshendra-converted.ppk (see [2])
    • Open Filezilla:
      • Edit, Settings..., SFTP, Add keyfile...
      • File, Site Manager, enter host of the EC2-instance (see chapter about SSH above) and enter the user (you can see the user when have logged in...)
      • Connect
    • On the command line by means of SCP ([3])

Force to HTTPS

If you support https it would be nice to force requests from http to forward to https. Unfortunately I was unable to do this even there exist 2 posts explaining this:

What I did:

  1. AWS Management Console, EC2, right-mouse-click 'Connect', copy ssh-command line
  2. execute ssh at the place where private key is situated
  3. cd /etc/httpd/sites/
  4. cp elasticbeanstalk ~
  5. sudo chmod 666 elasticbeanstal
    -> RW for Owner, Group and Other, see permissions calc tool
  6. vi elasticbeanstalk
    Place (or one of the other suggestions):
    RewriteEngine On
    RewriteCond %{HTTP:X-Forwarded-Proto} !https
    RewriteRule (.*) - [L]
    RewriteRule (.*) https://%{SERVER_NAME}%{REQUEST_URI} [L,R]
    before closing </VirtualHost>
  7. sudo chmod 644 elasticbeanstalk
  8. Restart application (necessary?)

=> No Luck!

Simple DB

Simple DB is a non-relational database available as web service. Links:

The access and secret key can be found in the AWS Management Console under the top right menu (press the triangle left to 'Help'), 'Security Credentials'. (Press 'Show' for the secret key.)

On my PC the ScratchPad can be used by following URL: file:///home/pfu/AWS/IAMCli-1.4.0/AmazonSimpleDB-2009-04-15-scratchpad/webapp/index.html. The credentials are locally saved under: /home/pfu/workspace/git/FwApps/FwLodurFfServicesExtractorUI/src/ch/peterfuerholz/fwapps/lodur/ffservicesextractor/ui.

DO MIND THAT THE USAGE OF SIMPLEDB IS RATHER EXPENSIVE!!!

Simple Storage Service (Amazon S3)

Access resources made public

Example:
Bucket-Name: neshendra.fw-tools
Key-Name: Smiley.png

URL: OK?
https://amazonaws.com.s3.amazonaws.com/fw-tools-res.neshendra.ch/huette.jpg Y (=default)
http://amazonaws.com.s3.amazonaws.com/fw-tools-res.neshendra.ch/huette.jpg Y
http://fw-tools-res.neshendra.ch.s3.amazonaws.com/huette.jpg Y

If you add following DNS-entry to your host-provider:

New URL-part: Fixed part: Type: Existing URL:
fw-tools-res .neshendra.ch CNAME fw-tools-res.neshendra.ch.s3.amazonaws.com

your resource is available as follows as well: http://fw-tools-res.neshendra.ch/huette.jpg

Copying files on the command line by means of S3cmd

  • Download from S3Tools
  • Unzip tar file and execute sudo python setup.py install
Configure it by s3cmd --configure
Commands (examples): s3cmd ls / s3cmd put -r /home/pfu/tmp s3://backup, sync instead put, ...

Use your own context.xml

According Amazon Forum you can place your context.xml next to web.xml and can thus configure e.g. session saving parameters in that way. Nevertheless it is available within /etc/tomcat6. (You probably need to stop the server and restart to configure it.)