Project Name | Stars | Downloads | Repos Using This | Packages Using This | Most Recent Commit | Total Releases | Latest Release | Open Issues | License | Language |
---|---|---|---|---|---|---|---|---|---|---|
Og Aws | 32,947 | 6 months ago | 148 | cc-by-4.0 | Shell | |||||
📙 Amazon Web Services — a practical guide | ||||||||||
Infracost | 9,156 | 6 hours ago | 129 | August 30, 2022 | 143 | apache-2.0 | Go | |||
Cloud cost estimates for Terraform in pull requests💰📉 Love your cloud bill! | ||||||||||
Aws Lambda Power Tuning | 4,490 | 14 days ago | 5 | apache-2.0 | JavaScript | |||||
AWS Lambda Power Tuning is an open-source tool that can help you visualize and fine-tune the memory/power configuration of Lambda functions. It runs in your own AWS account - powered by AWS Step Functions - and it supports three optimization strategies: cost, speed, and balanced. | ||||||||||
Opencost | 3,784 | 1 | 3 hours ago | 68 | September 19, 2022 | 141 | apache-2.0 | Go | ||
Cross-cloud cost allocation models for Kubernetes workloads | ||||||||||
Ice | 2,824 | 8 months ago | 109 | Java | ||||||
AWS Usage Tool | ||||||||||
Autospotting | 2,193 | 11 days ago | 43 | osl-3.0 | Go | |||||
Saves up to 90% of AWS EC2 costs by automating the use of spot instances on existing AutoScaling groups. Installs in minutes using CloudFormation or Terraform. Convenient to deploy at scale using StackSets. Uses tagging to avoid launch configuration changes. Automated spot termination handling. Reliable fallback to on-demand instances. | ||||||||||
Aws Well Architected Labs | 1,821 | 3 hours ago | 61 | apache-2.0 | Python | |||||
Hands on labs and code to help you learn, measure, and build using architectural best practices. | ||||||||||
Terratag | 789 | 24 days ago | 47 | July 13, 2022 | 2 | mpl-2.0 | Go | |||
Terratag is a CLI tool that enables users of Terraform to automatically create and maintain tags across their entire set of AWS, Azure, and GCP resources | ||||||||||
Finala | 641 | a year ago | 27 | other | Go | |||||
Finala is an open-source resource cloud scanner that analyzes, discloses, presents and notifies about wasteful and unused resources. | ||||||||||
Terraform Cost Estimation | 609 | a day ago | 8 | apache-2.0 | jq | |||||
Anonymized, secure, and free Terraform cost estimation based on Terraform plan (0.12+) or Terraform state (any version) |
Credits ∙ Contributing guidelines
Purpose
AWS in General
Specific AWS Services | Basics | Tips | Gotchas |
---|---|---|---|
ALB | 📗 | 📘 | 📙 |
AMIs | 📗 | 📘 | 📙 |
API Gateway | 📗 | 📘 | 📙 |
Auto Scaling | 📗 | 📘 | 📙 |
Batch | 📗 | 📘 | |
Certificate Manager | 📗 | 📘 | 📙 |
CLB (ELB) | 📗 | 📘 | 📙 |
CloudFront | 📗 | 📘 | 📙 |
CloudFormation | 📗 | 📘 | 📙 |
CloudWatch | 📗 | 📘 | 📙 |
Device Farm | 📗 | 📘 | 📙 |
DirectConnect | 📗 | 📘 | |
DynamoDB | 📗 | 📘 | 📙 |
EBS | 📗 | 📘 | 📙 |
EC2 | 📗 | 📘 | 📙 |
ECS | 📗 | 📘 | |
EKS | 📗 | 📘 | 📙 |
EFS | 📗 | 📘 | 📙 |
Elastic Beanstalk | 📗 | 📘 | 📙 |
Elastic IPs | 📗 | 📘 | 📙 |
ElastiCache | 📗 | 📘 | 📙 |
EMR | 📗 | 📘 | 📙 |
Fargate | 📗 | 📘 | 📙 |
Glacier | 📗 | 📘 | 📙 |
IoT | 📗 | 📘 | 📙 |
Kinesis Firehose | 📙 | ||
Kinesis Streams | 📗 | 📘 | 📙 |
KMS | 📗 | 📘 | 📙 |
Lambda | 📗 | 📘 | 📙 |
Load Balancers | 📗 | 📘 | 📙 |
Mobile Hub | 📗 | 📘 | 📙 |
OpsWorks | 📗 | 📘 | 📙 |
Quicksight | 📗 | 📙 | |
RDS | 📗 | 📘 | 📙 |
RDS Aurora | 📗 | 📘 | 📙 |
RDS Aurora MySQL | 📗 | 📘 | 📙 |
RDS Aurora PostgreSQL | 📗 | 📘 | 📙 |
RDS MySQL and MariaDB | 📗 | 📘 | 📙 |
RDS PostgreSQL | 📗 | 📘 | 📙 |
RDS SQL Server | 📗 | 📘 | 📙 |
Redshift | 📗 | 📘 | 📙 |
Route 53 | 📗 | 📘 | 📙 |
S3 | 📗 | 📘 | 📙 |
Security and IAM | 📗 | 📘 | 📙 |
SES | 📗 | 📘 | 📙 |
SNS | 📗 | 📘 | 📙 |
SQS | 📗 | 📘 | 📙 |
Step Functions | 📗 | 📘 | 📙 |
WAF | 📗 | 📘 | 📙 |
VPCs, Network Security, and Security Groups | 📗 | 📘 | 📙 |
Special Topics
Legal
Figures and Tables
A lot of information on AWS is already written. Most people learn AWS by reading a blog or a “getting started guide” and referring to the standard AWS references. Nonetheless, trustworthy and practical information and recommendations aren’t easy to come by. AWS’s own documentation is a great but sprawling resource few have time to read fully, and it doesn’t include anything but official facts, so omits experiences of engineers. The information in blogs or Stack Overflow is also not consistently up to date.
This guide is by and for engineers who use AWS. It aims to be a useful, living reference that consolidates links, tips, gotchas, and best practices. It arose from discussion and editing over beers by several engineers who have used AWS extensively.
Before using the guide, please read the license and disclaimer.
This is an early in-progress draft! It’s our first attempt at assembling this information, so is far from comprehensive still, and likely to have omissions or errors.
Please help by joining the Slack channel (we like to talk about AWS in general, even if you only have questions — discussion helps the community and guides improvements) and contributing to the guide. This guide is open to contributions, so unlike a blog, it can keep improving. Like any open source effort, we combine efforts but also review to ensure high quality.
There are now enough cloud and “big data” enterprise companies and products that few can keep up with the market landscape.
We’ve assembled a landscape of a few of the services. This is far from complete, but tries to emphasize services that are popular with AWS practitioners — services that specifically help with AWS, or a complementary, or tools almost anyone using AWS must learn.
🚧 Suggestions to improve this figure? Please file an issue.
Many services within AWS can at least be compared with Google Cloud offerings or with internal Google services. And often times you could assemble the same thing yourself with open source software. This table is an effort at listing these rough correspondences. (Remember that this table is imperfect as in almost every case there are subtle differences of features!)
Service | AWS | Google Cloud | Google Internal | Microsoft Azure | Other providers | Open source “build your own” | Openstack |
---|---|---|---|---|---|---|---|
Virtual server | EC2 | Compute Engine (GCE) | Virtual Machine | DigitalOcean | OpenStack | Nova | |
PaaS | Elastic Beanstalk | App Engine | App Engine | Web Apps | Heroku, AppFog, OpenShift | Meteor, AppScale, Cloud Foundry, Convox | |
Serverless, microservices | Lambda, API Gateway | Functions | Function Apps | PubNub Blocks, Auth0 Webtask | Kong, Tyk | Qinling | |
Container, cluster manager | ECS, EKS, Fargate | Container Engine, Kubernetes | Borg or Omega | Container Service | Kubernetes, Mesos, Aurora | Zun | |
Object storage | S3 | Cloud Storage | GFS | Storage Account | DigitalOcean Spaces | Swift, HDFS, Minio | Swift |
Block storage | EBS | Persistent Disk | Storage Account | DigitalOcean Volumes | NFS | Cinder | |
SQL datastore | RDS | Cloud SQL | SQL Database | MySQL, PostgreSQL | Trove (stores NoSQL as well) | ||
Sharded RDBMS | Cloud Spanner | F1, Spanner | Azure Database for PostgreSQL - Hyperscale (Citus) | Crate.io, CockroachDB | |||
Bigtable | Cloud Bigtable | Bigtable | HBase | ||||
Key-value store, column store | DynamoDB | Cloud Datastore | Megastore | Tables, DocumentDB | Cassandra, CouchDB, RethinkDB, Redis | ||
Memory cache | ElastiCache | App Engine Memcache | Redis Cache | Memcached, Redis | |||
Search | CloudSearch, Elasticsearch (managed) | Search | Algolia, QBox, Elastic Cloud | Elasticsearch, Solr | |||
Data warehouse | Redshift | BigQuery | Dremel | SQL Data Warehouse | Oracle, IBM, SAP, HP, many others | Greenplum | |
Business intelligence | QuickSight | Data Studio 360 | Power BI | Tableau | |||
Lock manager | DynamoDB (weak) | Chubby | Lease blobs in Storage Account | ZooKeeper, Etcd, Consul | |||
Message broker | SQS, SNS, IoT | Pub/Sub | PubSub2 | Service Bus | RabbitMQ, Kafka, 0MQ | ||
Streaming, distributed log | Kinesis | Dataflow | PubSub2 | Event Hubs | Kafka Streams, Apex, Flink, Spark Streaming, Storm | ||
MapReduce | EMR | Dataproc | MapReduce | HDInsight, DataLake Analytics | Qubole | Hadoop | |
Monitoring | CloudWatch | Stackdriver Monitoring | Borgmon | Monitor | Prometheus(?) | ||
Tracing | X-Ray | Stackdriver Trace | Monitor (Application Insights) | DataDog, New Relic, Epsagon | Zipkin, Jaeger, Appdash | ||
Metric management | Borgmon, TSDB | Application Insights | Graphite, InfluxDB, OpenTSDB, Grafana, Riemann, Prometheus | ||||
CDN | CloudFront | Cloud CDN | CDN | Akamai, Fastly, Cloudflare, Limelight Networks | Apache Traffic Server | ||
Load balancer | CLB/ALB | Load Balancing | GFE | Load Balancer, Application Gateway | nginx, HAProxy, Apache Traffic Server | ||
DNS | Route53 | DNS | DNS | bind | |||
SES | Sendgrid, Mandrill, Postmark | ||||||
Git hosting | CodeCommit | Cloud Source Repositories | Visual Studio Team Services | GitHub, BitBucket | GitLab | ||
User authentication | Cognito | Firebase Authentication | Azure Active Directory | oauth.io | |||
Mobile app analytics | Mobile Analytics | Firebase Analytics | HockeyApp | Mixpanel | |||
Mobile app testing | Device Farm | Firebase Test Lab | Xamarin Test Cloud | BrowserStack, Sauce Labs, Testdroid | |||
Managing SSL/TLS certificates | Certificate Manager | Let's Encrypt, Comodo, Symantec, GlobalSign | |||||
Automatic speech recognition and natural language understanding | Transcribe (ASR), Lex (NLU) | Cloud Speech API, Natural Language API | Cognitive services | AYLIEN Text Analysis API, Ambiverse Natural Language Understanding API | Stanford's Core NLP Suite, Apache OpenNLP, Apache UIMA, spaCy | ||
Text-to-speech engine in the cloud | Polly | Nuance, Vocalware, IBM | Mimic, eSpeak, MaryTTS | ||||
Image recognition | Rekognition | Vision API | Cognitive services | IBM Watson, Clarifai | TensorFlow, OpenCV | ||
OCR (Text recognition) | Textract (documents), Rekognition (photographs) | Cloud Vision API | Computer Vision API | Tesseract | |||
Language Translation | Translate | Translate | Translator Text API | Apertium | |||
File Share and Sync | WorkDocs | Google Docs | OneDrive | Dropbox, Box, Citrix File Share | ownCloud | ||
Machine Learning | SageMaker, DeepLens, ML | ML Engine, Auto ML | ML Studio | Watson ML | |||
Data Loss Prevention | Macie | Cloud Data Loss Prevention | Azure Information Protection |
🚧 Please help fill this table in.
Selected resources with more detail on this chart:
It’s important to know the maturity of each AWS product. Here is a mostly complete list of first release date, with links to the release notes. Most recently released services are first. Not all services are available in all regions; see this table.
Service | Original release | Availability | CLI Support | HIPAA Compliant | PCI-DSS Compliant |
---|---|---|---|---|---|
🐥X-Ray | 2016-12 | General | ✓ | ✓ | ✓ |
🐥Lex | 2016-11 | Preview | |||
🐥Polly | 2016-11 | General | ✓ | ✓ | ✓ |
🐥Rekognition | 2016-11 | General | ✓ | ✓ | ✓ |
🐥Athena | 2016-11 | General | ✓ | ✓ | ✓ |
🐥Batch | 2016-11 | General | ✓ | ✓ | ✓ |
🐥Database Migration Service | 2016-03 | General | ✓ | ✓ | |
🐥Certificate Manager | 2016-01 | General | ✓ | ✓ | ✓ |
🐥IoT | 2015-08 | General | ✓ | ✓ | ✓13 |
🐥WAF | 2015-10 | General | ✓ | ✓ | ✓ |
🐥Data Pipeline | 2015-10 | General | ✓ | ||
🐥Elasticsearch | 2015-10 | General | ✓ | ✓ | ✓ |
🐥Aurora | 2015-07 | General | ✓ | ✓3 | ✓3 |
🐥Service Catalog | 2015-07 | General | ✓ | ✓ | ✓ |
🐥Device Farm | 2015-07 | General | ✓ | ||
🐥CodePipeline | 2015-07 | General | ✓ | ✓ | |
🐥CodeCommit | 2015-07 | General | ✓ | ✓ | ✓ |
🐥API Gateway | 2015-07 | General | ✓ | ✓1 | ✓ |
🐥Config | 2015-06 | General | ✓ | ✓ | ✓ |
🐥EFS | 2015-05 | General | ✓ | ✓ | ✓ |
🐥Machine Learning | 2015-04 | General | ✓ | ||
Lambda | 2014-11 | General | ✓ | ✓ | ✓ |
ECS | 2014-11 | General | ✓ | ✓ | ✓ |
EKS | 2018-06 | General | ✓12 | ✓ | ✓ |
KMS | 2014-11 | General | ✓ | ✓ | ✓ |
CodeDeploy | 2014-11 | General | ✓ | ✓ | |
Kinesis | 2013-12 | General | ✓ | ✓ | ✓11 |
CloudTrail | 2013-11 | General | ✓ | ✓ | ✓ |
AppStream | 2013-11 | Preview | ✓ | ||
CloudHSM | 2013-03 | General | ✓ | ✓ | ✓ |
Silk | 2013-03 | Obsolete? | |||
OpsWorks | 2013-02 | General | ✓ | ✓ | ✓ |
Redshift | 2013-02 | General | ✓ | ✓ | ✓ |
Elastic Transcoder | 2013-01 | General | ✓ | ||
Glacier | 2012-08 | General | ✓ | ✓ | ✓ |
CloudSearch | 2012-04 | General | ✓ | ||
SWF | 2012-02 | General | ✓ | ✓ | ✓ |
Storage Gateway | 2012-01 | General | ✓ | ✓ | ✓ |
DynamoDB | 2012-01 | General | ✓ | ✓ | ✓ |
DirectConnect | 2011-08 | General | ✓ | ✓ | ✓ |
ElastiCache | 2011-08 | General | ✓ | ✓14 | ✓14 |
CloudFormation | 2011-04 | General | ✓ | ✓ | ✓ |
SES | 2011-01 | General | ✓ | ✓ | |
Elastic Beanstalk | 2010-12 | General | ✓ | ✓ | ✓ |
Route 53 | 2010-10 | General | ✓ | ✓ | ✓ |
IAM | 2010-09 | General | ✓ | ✓ | |
SNS | 2010-04 | General | ✓ | ✓ | ✓ |
EMR | 2010-04 | General | ✓ | ✓ | ✓ |
RDS | 2009-12 | General | ✓ | ✓2 | ✓9 |
VPC | 2009-08 | General | ✓ | ✓ | ✓ |
Snowball | 2015-10 | General | ✓ | ✓ | ✓15 |
Snowmobile | 2016-11 | General | ✓ | ✓ | |
CloudWatch | 2009-05 | General | ✓ | ✓ | ✓ |
CloudFront | 2008-11 | General | ✓ | ✓4 | ✓ |
Fulfillment Web Service | 2008-03 | Obsolete? | |||
SimpleDB | 2007-12 | ❗Nearly obsolete | ✓ | ✓ | |
DevPay | 2007-12 | General | |||
Flexible Payments Service | 2007-08 | Retired | |||
EC2 | 2006-08 | General | ✓ | ✓5,6,7 | ✓6,7,10 |
SQS | 2006-07 | General | ✓ | ✓ | ✓ |
S3 | 2006-03 | General | ✓ | ✓8 | ✓ |
Alexa Top Sites | 2006-01 | General ❗HTTP-only | |||
Alexa Web Information Service | 2005-10 | General ❗HTTP-only |
1: Excludes use of Amazon API Gateway caching
2: RDS MySQL, Oracle, and PostgreSQL engines only
3: MySQL-compatible Aurora edition only
4: Excludes [email protected]
5: Includes EC2 Systems Manager
6: Includes Elastic Block Storage (EBS)
7: Includes Elastic Load Balancing
8: Includes S3 Transfer Acceleration
9: Includes RDS MySQL, Oracle, PostgreSQL, SQL Server, and MariaDB
10: Includes Auto-Scaling
11: Data Analytics, Streams, Video Streams and Firehose
12: Kubernetes uses a custom CLI for Pod/Service management called kubectl. AWS CLI only handles Kubernetes Master concerns
13: IoT Core (includes Device Management) and Greengrass
14: ElastiCache for Redis only
15: Snowball and Snowball Edge
Associate level certifications were once required as pre-requisites to taking the Professional examinations - this is no longer the case.
Certifications are required to access certificate lounges at official AWS events such as Summits and re:Invent. Lounges typically provide power charging points, seats and relatively better coffee.
A great challenge in using AWS to build complex systems (and with DevOps in general) is to manage infrastructure state effectively over time. In general, this boils down to three broad goals for the state of your infrastructure:
Much of what we discuss below is really about how to improve the answers to these questions.
There are several approaches to deploying infrastructure with AWS, from the console to complex automation tools, to third-party services, all of which attempt to help achieve visibility, automation, and flexibility.
The first way most people experiment with AWS is via its web interface, the AWS Console. But using the Console is a highly manual process, and often works against automation or flexibility.
So if you’re not going to manage your AWS configurations manually, what should you do? Sadly, there are no simple, universal answers — each approach has pros and cons, and the approaches taken by different companies vary widely, and include directly using APIs (and building tooling on top yourself), using command-line tools, and using third-party tools and services.
This guide is about AWS, not DevOps or server configuration management in general. But before getting into AWS in detail, it’s worth noting that in addition to the configuration management for your AWS resources, there is the long-standing problem of configuration management for servers themselves.
We cover security basics first, since configuring user accounts is something you usually have to do early on when setting up your system.
http://s3.amazonaws.com/bucket-name/filename
. No authorization or signature is required to access data in this category.s3:PutObjectAcl
or s3:PutObjectVersionAcl
permissions.<bucket_name>.s3-us-east-1.amazonaws.com
, as long as the name is DNS compliant.https://BUCKET.s3.amazonaws.com
. However, as of Aug 11, 2016 it now supports both IPv4 & IPv6! To use both, you have to enable dualstack either in your preferred API client or by directly using this url scheme https://BUCKET.s3.dualstack.REGION.amazonaws.com
. This extends to S3 Transfer Acceleration as well.As an illustration of comparative features and price, the table below gives S3 Standard, RRS, IA, in comparison with Glacier, EBS, EFS, and EC2 d2.xlarge instance store using Virginia region as of Sept 2017.
Durability (per year) | Availability “designed” | Availability SLA | Storage (per TB per month) | GET or retrieve (per million) | Write or archive (per million) | |
---|---|---|---|---|---|---|
Glacier | Eleven 9s | Sloooow | – | $4 | $50 | $50 |
S3 IA | Eleven 9s | 99.9% | 99% | $12.50 | $1 | $10 |
S3 Standard | Eleven 9s | 99.99% | 99.9% | $23 | $0.40 | $5 |
EBS | 99.8% | Unstated | 99.99% | $25/$45/$100/$125+ (sc1/st1/gp2/io1) | ||
EFS | “High” | “High” | – | $300 | ||
EC2 d2.xlarge instance store | Unstated | Unstated | – | $25.44 | $0 | $0 |
Especially notable items are in boldface. Sources: S3 pricing, S3 SLA, S3 FAQ, RRS info (note that this is considered deprecated), Glacier pricing, EBS availability and durability, EBS pricing, EFS pricing, EC2 SLA
🔹Picking regions: When you first set up, consider which regions you want to use first. Many people in North America just automatically set up in the us-east-1 (N. Virginia) region, which is the default, but it’s worth considering if this is best up front. You'll want to evaluate service availability (some services are not available in all regions), costing (baseline costs also vary by region by up to 10-30% (generally lowest in us-east-1 for comparison purposes)), and compliance (various countries have differing regulations with regard to data privacy, for example).
Instance types: EC2 instances come in many types, corresponding to the capabilities of the virtual machine in CPU architecture and speed, RAM, disk sizes and types (SSD or magnetic), and network bandwidth.
Turn off your instances when they aren’t in use. For many situations such as testing or staging resources, you may not need your instances on 24/7, and you won’t need to pay EC2 running costs when they are suspended. Given that costs are calculated based on usage, this is a simple mechanism for cost savings. This can be achieved using Lambda and CloudWatch, deploying the Instance Scheduler solution, an open source option like cloudcycler, or a SaaS provider like GorillaStack. (Note: if you turn off instances with an ephemeral root volume, any state will be lost when the instance is turned off. Therefore, for stateful applications it is safer to turn off EBS backed instances).
Dedicated instances and dedicated hosts are assigned hardware, instead of usual virtual instances. They are more expensive than virtual instances but can be preferable for performance, compliance, financial modeling, or licensing reasons.
32 bit vs 64 bit: A few micro, small, and medium instances are still available to use as 32-bit architecture. You’ll be using 64-bit EC2 (“amd64”) instances nowadays, though smaller instances still support 32 bit (“i386”). Use 64 bit unless you have legacy constraints or other good reasons to use 32.
HVM vs PV: There are two kinds of virtualization technology used by EC2, hardware virtual machine (HVM) and paravirtual (PV). Historically, PV was the usual type, but now HVM is becoming the standard. If you want to use the newest instance types, you must use HVM. See the instance type matrix for details.
Operating system: To use EC2, you’ll need to pick a base operating system. It can be Windows or Linux, such as Ubuntu or Amazon Linux. You do this with AMIs, which are covered in more detail in their own section below.
Limits: You can’t create arbitrary numbers of instances. Default limits on numbers of EC2 instances per account vary by instance type, as described in this list.
❗Use termination protection: For any instances that are important and long-lived (in particular, aren't part of auto-scaling), enable termination protection. This is an important line of defense against user mistakes, such as accidentally terminating many instances instead of just one due to human error.
SSH key management:
GPU support: You can rent GPU-enabled instances on EC2 for use in machine learning or graphics rendering workloads.
All current EC2 instance types can take advantage of IPv6 addressing, so long as they are launched in a subnet with an allocated CIDR range in an IPv6-enabled VPC.
PutMetricData
API, StorageResolution
is an attribute of each item you send in the MetricData
array, not a direct parameter of the PutMetricData
API call.🔸Amazon Linux package versions: By default, instances based on Amazon Linux AMIs are configured point to the latest versions of packages in Amazon’s package repository. This means that the package versions that get installed are not locked and it is possible for changes, including breaking ones, to appear when applying updates in the future. If you bake your AMIs with updates already applied, this is unlikely to cause problems in running services whose instances are based on those AMIs – breaks will appear at the earlier AMI-baking stage of your build process, and will need to be fixed or worked around before new AMIs can be generated. There is a “lock on launch” feature that allows you to configure Amazon Linux instances to target the repository of a particular major version of the Amazon Linux AMI, reducing the likelihood that breaks caused by Amazon-initiated package version changes will occur at package install time but at the cost of not having updated packages get automatically installed by future update runs. Pairing use of the “lock on launch” feature with a process to advance the Amazon Linux AMI at your discretion can give you tighter control over update behaviors and timings.
Cloud-Init Defaults: Oftentimes users create AMIs after performing customizations (albeit manually or via some tool such as Packer or Ansible). If you're not careful to alter cloud-init settings that correspond to the system service (e.g. sshd, etc.) you've customized, you may find that your changes are no longer in effect after booting your new AMI for the first time, as cloud-init has overwritten them.
Some distros have different files than others, but all are generally located in /etc/cloud/
, regardless of distro. You will want to review these files carefully for your chosen distro before rolling your own AMIs. A complete reference to cloud-init is available on the cloud-init site. This is an advanced configuration mechanism, so test any changes made to these files in a sandbox prior to any serious usage.
dd
or fio
as per the official documentation.With EFS being based on NFSv4.1, any directory on the EFS can be mounted directly, it doesn't have to be the root directory. One application could mount fs-12345678:/prog1, another fs-12345678:/prog2.
User and group level permissions can be used to control access to certain directories on the EFS file system.
⏱ Sharing EFS filesystems: One EFS filesystem can be used for multiple applications or services, but it should be considered carefully:
Pros:
Cons:
find
or chown -R
can have an adverse impact on performance.update-certificate
call with the following process:Deployment policy
to All at once
eb config save --cfg myEBConfig
Name
tag.elasticbeanstalk/saved_configs/
, be aware that this is not kept in sync with the EB environment config. You'll need to manually fetch and save for changes to take effectAWS-DUOSECURITYINC
. This is a best effort request, expect a denial.aws-quicksight-service-role-v0
does not have the necessary permissions. You can't pick another role to be used but you can add more permissions to the role. The error message you will receive is an SQL_EXCEPTION
with details SYNTAX_ERROR: line 2:8: Column 'columnname' cannot be resolved
where columnname
is the first column in your table. RDS offers MySQL versions 5.5, 5.6, 5.7 and 5.8.
RDS offers MariaDB versions 10.0, 10.1, 10.2 and 10.3.
rds_superuser
that can do most of the needed operations but there are some limitations.Aurora is a cloud only database service designed to provide a distributed, fault-tolerant relational database with self-healing storage and auto-scaling up to 64TB per instance. It currently comes in two versions, a MySQL compatible system, and a PostgreSQL compatible system.
awslogs
for CloudWatch (make sure a group is made for the logs first). Drivers such as fluentd are not enabled by default. You can, install the agent and enable the driver by adding ECS_AVAILABLE_LOGGING_DRIVERS='["awslogs","fluentd"]'
to /etc/ecs/ecs.config
.ECS_ENGINE_TASK_CLEANUP_WAIT_DURATION=10m
and ECS_IMAGE_CLEANUP_INTERVAL=10m
to /etc/ecs/ecs.config
. More information on optimizing ECS disk cleanup.🚧 Please help expand this incomplete section.
1: https://docs.aws.amazon.com/eks/latest/userguide/create-kubeconfig.html
2: https://aws.amazon.com/about-aws/whats-new/2019/05/amazon-eks-simplifies-kubernetes-cluster-authentication/
🚧 Please help expand this incomplete section.
Execution failed due to an internal error
. One possible reason for this error is that even though your backend server is up and running, it may be doing something outside of the HTTP specification (like not sending well-formed chunked messages). You can test by hitting your backend directly with the curl --raw -S -i <backend-endpoint-url>
and seeing if it complains.🚧 Please help expand this incomplete section.
validate-template
.DescribeStack
API calls, and get imported to other Stacks as part of the recent addition of cross-stack references.
-Note that importing outputs in a stack from another stack creates a hard dependency that is tracked by CloudFormation. You will not be able to delete the stack with the outputs until there are no importing stacks.CreateDeployment
API has a default limit of 3 requests per minute as of 1/12/2018. This limit is readily exceeded even in moderately-sized CloudFormation stacks. Creating CW alarms is another commonly seen limit (PutMetricAlarm
, 3 tps as of 1/12/2018) especially when creating many autoscaling policies for DynamoDB. One way to work around this limit is to include CloudFormation 'DependsOn' clauses to artificially chain resource creation.❗Security groups are your first line of defense for your servers. Be extremely restrictive of what ports are open to all incoming connections. In general, if you use CLBs, ALBs or other load balancing, the only ports that need to be open to incoming traffic would be port 22 and whatever port your application uses. Security groups access policy is 'deny by default'.
Port hygiene: A good habit is to pick unique ports within an unusual range for each different kind of production service. For example, your web frontend might use 3010, your backend services 3020 and 3021, and your Postgres instances the usual 5432. Then make sure you have fine-grained security groups for each set of servers. This makes you disciplined about listing out your services, but also is more error-proof. For example, should you accidentally have an extra Apache server running on the default port 80 on a backend server, it will not be exposed.
Migrating from Classic: For migrating from older EC2-Classic deployments to modern EC2-VPC setup, this article may be of help.
For basic AWS use, one default VPC may be sufficient. But as you scale up, you should consider mapping out network topology more thoroughly. A good overview of best practices is here.
Consider controlling access to you private AWS resources through a VPN.
🔹Consider using other security groups as sources for security group rules instead of using CIDRs — that way, all hosts in the source security group and only hosts in that security group are allowed access. This is a much more dynamic and secure way of managing security group rules.
VPC Flow Logs allow you to monitor the network traffic to, from, and within your VPC. Logs are stored in CloudWatch Logs groups, and can be used for security monitoring (with third party tools), performance evaluation, and forensic investigation.
IPv6 is available in VPC. Along with this announcement came the introduction of the Egress-Only Internet Gateway. In cases where one would use NAT Gateways to enable egress-only traffic for their VPC in IPv4, one can use an Egress-Only Internet Gateway for the same purpose in IPv6.
Amazon provides an IPv6 CIDR block for your VPC at your request - at present you cannot implement your own IPv6 block if you happen to own one already.
New and existing VPCs can both use IPv6. Existing VPCs will need to be configured to have an IPv6 CIDR block associated with them, just as new VPCs do.
NOT NULL
column constraints are enforced. See here for more information on defining constraints.awsmobile {pull|push}
commands, to sync from cloud to folder, and back again.*.bar.example.com
. This would be valid for foo.bar.example.com
but not bar.example.com
. Likewise it would also not be valid for www.bar.foo.example.com
. You would need to add each of these domains to the certificate request."I acknowledge at the moment, there is no method to add or remove a name from a certificate. Instead, you must request a new certificate with the revised namelist and you must then re-approve all of the names in the certificate, even if they'd been previously approved."
This section covers tips and information on achieving high availability.
This section covers a few unusually useful or “must know about” resources or lists.
The authors and contributors to this content cannot guarantee the validity of the information found here. Please make sure that you understand that the information provided here is being provided freely, and that no kind of agreement or contract is created between you and any persons associated with this content or project. The authors and contributors do not assume and hereby disclaim any liability to any party for any loss, damage, or disruption caused by errors or omissions in the information contained in, associated with, or linked from this content, whether such errors or omissions result from negligence, accident, or any other cause.
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.