Cloud → Google Cloud Platform → Associate Engineer → Compute Engine
Compute Engine
Google Cloud Platform (GCP) – Master yourself in 30 Days
By Navin Gupta
Email: navingcp11@gmail.com WhatsApp group: https://chat.whatsapp.com/CT6hN4r61hJLXevuyChBVJ
Table of Contents
Google Cloud Platform
DAY 2: 4. Introduction – Compute Engine
4.1 Compute Engine - Type
1.2 Cloud Service Model or Type
1.3 Google Cloud Platform: Region and Zones
1.4 Resource hierarchy
2. Goole Cloud Platform – Billing
2.1 Cost Management
2.1.1 Report
2.1.2 Cost Table
2.1.3 Cost Breakdown
2.1.4. Budget and Alerts
2.1.5 Billing Export
2.1.6 Anomalies
2.2 Cost Optimization
2.2.1 FinOps Hub
2.2.2 Committed Use discount
2.2.3 Committed use discount analysis (CUD Analysis)
2.2.4 Pricing, Cost estimation and Credits
2.3. Payments
2.4. Billing Management
Index
4.1 Introduction: Compute Engine is an infrastructure as a service (IaaS) service that allows users to create a virtual machine on top of the KVM hypervisor. An user can select configuration for virtual machine which is needed to run a business like operating system, storage, memory, computing power, networking, security etc.
4.2 Compute Engine Google Cloud Platform - Benefit: Compute engine in GCP offers following benefits:
Cost
: You only pay for the computing time you use, and GCP calculates how much computing power you need.
Scalability
: You can easily scale
your resources up or down based on your
business
needs.
Storage
:
You can p
rovision
internal memory of
between 10 and 65536 GB
as well as same you can attach external disk as storage for the same size
.
Stability
: GCE provides live migration of VMs between hosts, which makes services more stable.
Backups
: GCP has a robust, redundant backup system that GCE uses for its flagship products.
Security
: GCE is a secure place for cloud applications.
Machine learning
: GCP provides access to machine learning data, which can help companies improve their products.
Speed and reliability
: Google invests in upgrading its hardware and software, and creating new data
cent
er
around the world.
Integrity monitoring
: You can monitor and verify the runtime boot integrity of your shielded VM instances.
4.3 Google Cloud Platform: Machine Family
4.3.1 General Purpose: Best price-performance ratio for a variety of workloads. It has 2 machine type. These instance types are used when there are no specific requirements and a balance of compute and memory resources is required. They offer the best price-to-performance ratio for a variety of workloads.
Predefined:
In the category, Google has created the series of machine where predefined vCPU and memory has been allocated.
Following table shows the list the machine series.
|
Series |
Description |
vCPUs |
Memory |
Platform |
|
C4 |
Consistently high performance |
2 - 192 |
4 – 1,488 GB |
Intel Emerald Rapids |
|
N4 |
Flexible and cost-optimised |
Feb-80 |
4 – 640 GB |
Intel Emerald Rapids |
|
C3 |
Consistently high performance |
4 - 192 |
8 – 1,536 GB |
Intel Sapphire Rapids |
|
C3D |
Consistently high performance |
4 - 360 |
8 – 2,880 GB |
AMD Genoa |
|
E2 |
Low-cost day-to-day computing |
0.25 - 32 |
1 – 128 GB |
Based on availability |
|
N2 |
Balanced price and performance |
2 - 128 |
2 – 864 GB |
Intel Cascade and Ice Lake |
|
N2D |
Balanced price and performance |
2 - 224 |
2 – 896 GB |
AMD EPYC |
|
T2A |
Scale-out workloads |
Jan-48 |
4 – 192 GB |
Ampere Altra ARM |
|
T2D |
Scale-out workloads |
Jan-60 |
4 – 240 GB |
AMD EPYC Milan |
|
N1 |
Balanced price and performance |
0.25 - 96 |
0.6 – 624 GB |
Intel Skylake |
Custom:
In this type, user can create a machine for his workload as per need.
He can select vCPU and memory. Even memory can be given in decimal
4.3.2 Compute Optimized: Highest performance per core on Compute Engine and optimized for compute-intensive workloads.
|
Series |
Description |
vCPUs |
Memory |
Platform |
|
H3 |
High-performance computing workloads |
Mar-00 |
352 GB |
Intel Sapphire Rapids |
|
C2 |
Ultra-high performance, compute-intensive workloads |
22007 |
16 – 240 GB |
Intel Cascade Lake |
|
C2D |
Ultra-high performance, compute-intensive workloads |
2 - 112 |
4 – 896 GB |
AMD EPYC Milan |
4.3.3 Memory Optimized: Ideal for memory-intensive workloads, offering more memory per core than other machine families, with up to 12 TB of memory.
|
Series |
Description |
vCPUs |
Memory |
Platform |
|
X4 |
Extra-large, in-memory databases |
960 - 1920 |
16,384 – 32,768 GB |
Intel Sapphire Rapids |
|
M3 |
High memory, memory-intensive workloads |
32 - 128 |
976 – 3,904 GB |
Intel Ice Lake |
|
M2 |
Ultra-high memory, in-memory databases |
208 - 416 |
5,888 – 11,776 GB |
Intel Cascade Lake |
|
M1 |
High memory, memory-intensive workloads |
40 - 160 |
961 – 3,844 GB |
Intel Skylake |
4.3.4 Storage Optimized: Best for workloads that are low in core usage and high in storage density.
|
Series |
Description |
vCPUs |
Memory |
Attached storage |
Platform |
|
Z3 |
High-density storage |
88 - 176 |
704 – 1,408 GB |
36,000 GiB |
Intel Sapphire Rapids |
4.3.5 Accelerate optimized (GPUs Based Machine): Ideal for massively parallelized Compute Unified Device Architecture (CUDA) compute workloads, such as machine learning (ML) and high-performance computing (HPC). This family is the best option for workloads that require GPUs.
Predefined: In the category, Google has created the series of machine where predefined vCPU and memory has been allocated. Following table shows the list the machine series.
Custom: In this type, user can create a machine for his workload as per need. He can select vCPU and memory. Even memory can be given in decimal
4.4 Compute Engine – Name:
Must be unique within a project and zone.
Can contain lowercase letters, numbers, and hyphens.
Must start with a letter and can be 1-63 characters long.
4.5 Compute Engine – Region and Zone
Compute Engine is zonal service
At least one region and zone must be selected while creating the compute engine
It
cannot
be migrated
from one zone to another zone
.
Once
zone is selected, it
cannot
be changed later
4.6 Compute Engine – Availability Policies: Compute Engine VMs can use either the standard provisioning model (standard VMs) (default) or the spot provisioning model (Spot VMs).
|
S.No. |
Feature |
Standard VMs |
Spot VMs |
|
1 |
Cost |
Regular pricing |
60-80% lower cost |
|
2 |
Uptime Guarantee |
Guaranteed until you stop it |
Can be terminated anytime by GCP |
|
3 |
Pre-emption |
No pre-emption |
Yes, pre-empted when resources are needed |
|
4 |
Maximum Lifetime |
Unlimited |
24 hours max, but can terminate sooner |
|
5 |
Ideal Use Case |
Long-running, critical applications |
Fault-tolerant, non-critical, batch jobs |
|
6 |
Live Migration |
Yes (during maintenance) |
No |
|
7 |
Automatic Restart |
Yes (if set in policies) |
No |
|
8 |
Usage |
Production workloads, databases, web servers |
Batch processing, CI/CD, testing, rendering |
4.7 Compute Engine – Operating System: Use operating system (OS) images to create boot disks for your virtual machine (VM) instances. You can use one of the following OS image types:
are provided and maintained by Google,
open-source
communities, and third-party vendors. By default, all Google Cloud projects have access to these OS images and can use them to
.
are available only to your Google Cloud project. You can
from boot disks and other images. Then, use the custom OS image to
.
Use custom OS images for the following tasks:
Snapshot
Archive Snapshot
Existing Disk
Some OS images are also capable of running containers on Compute Engine.
4.8 Compute Engine – External Disk or attach Existing Disk: By default, each Compute Engine VM has a single boot disk that contains the operating system. You can also attach external disk as per your need.
Min size – 10 GB (for both)
Max Size -
65536 GB
Deletion Rule:
Keep Disk after deleting VM
Delete Disk after deleting VM
Mode:
Read/Write
Read Only
Location of disk:
Zonal
Regional
Replica Zone
Question: Can you attach an external disk to virtual machine which is in different zone or region?
Persistent Disk types: There are four types of disks you can attach to your virtual machine.
When you configure a persistent disk, you can select one of the following disk types:
Balanced persistent disks (pd-balanced)
An alternative to performance (pd-
ssd
) persistent disks
Balance of performance and cost. For most VM shapes, except very large ones, these disks have the same maximum IOPS as SSD persistent disks and lower IOPS per
GiB.
This disk type offers performance levels suitable for most general-purpose applications at a price point between that of standard and performance (pd-
ssd
) persistent disks.
Backed by solid-state drives (SSD).
Performance (SSD) persistent disks (pd-
ssd
)
Suitable for enterprise applications and high-performance databases that require lower latency and more IOPS than standard persistent disks provide.
Backed by solid-state drives (SSD).
Standard persistent disks (pd-standard)
Suitable for large data processing workloads that primarily use sequential I/
Os
.
Backed by standard hard disk drives (HDD).
Extreme persistent disks (pd-extreme)
Offer consistently high performance for both random access workloads and bulk throughput.
Designed for high-end database workloads.
Allow you to provision the target IOPS.
Backed by solid-state drives (SSD).
Available with a limited number of
.
4.9 Compute Engine – Back Up Plan: To take a backup user should have necessary access. A backup plan has backup rules, where the following applies:
One or more backup rules can be used.
You can define the frequency for backup creation—hourly, daily, weekly, monthly, or yearly.
For weekly backups, you can choose a weekday for the rule.
For monthly backups, you can choose a specific day of the month for the rule. For example, the 15th of the month.
You can use for both scheduled or on-demand backups.
Includes a backup window where you can define the specific timeframe of when backup jobs can start. The backup window uses the following:
24-hour clock format, with start and end times between 00 and 24 hours.
A minimum of six hours for the window.
Backup plan and rule names
Your backup plan names and rule names must meet the following requirements:
Contain lowercase letters, numeric characters, dashes (-), underscores (_), and periods (.), spaces are not allowed
Start and end with a number or letter
Maximum of 63 characters
Cannot be represented as an IP address in dotted-decimal notation. For example, 192.0.2.255
Create a backup plan
In the Google Cloud console, go to the Backup plans page.
Click Create Backup plan.
In the Backup plan name field, enter a name for the backup plan. You can't change the name of a plan after the backup plan is created.
In the Backup plan description field, enter an optional description for the backup plan.
From the Regions list, select a backup plan region. The backup plan is created in this region. You can protect resources in the same region as the backup plan.
From the Backup vault list, select a backup vault to store the backups.
Click Add rule.
In the Add a backup rule pane, enter your backup rule information and click Save.
Name your backup rule: Enter a name for the backup rule.
Choose when to create backups: Specify the recurrence and frequency of the backup.
Window: Select the
Timezone
, Start time, and
Endtime
for the backup job.
Choose how long backups are kept before they are deleted: Enter the duration in days that the backups should be retained before they are deleted. Note that this value must be equal to or greater than the
backup vault minimum enforced retention period
.
Click Create.
4.10 Compute Engine Price: Compute engine pricing depends on-
Region
Storage
vCPU
Memory
Storage
Static I
p
s
Operating System
Usage Duration
4.11 Compute Engine – Networking: Every VM is part of a VPC network. VPC networks provide connectivity for your VM instance to other Google Cloud products and to the internet. VPC networks can be auto mode or custom mode.
Firewall
-
F
irewall rules are used to allow or deny traffic based on specified conditions.
You can enable traffic based on your requirement as shown below.
IP Forwarding
-
IP Forwarding in Google Cloud Platform (GCP) allows virtual machine (VM) instances to forward packets that are not destined for their own IP addresses.
Network bandwidth
–
It r
efers to the maximum rate of data transfer across a network connection.
Maximum outbound network bandwidth: 2Gbps
VM to Public IP: 2Gbps
Network interfaces
– It
allow
s
virtual machine (VM) instances to connect to networks. They define how a VM communicates with other resources, both within GCP and externally.
It has a default network when no VPC is defined.
IP stack type
-
IPv4 (single-stack)
and
Pv4 and IPv6 (dual-stack)
|
Feature |
IPv4 (Single-Stack) |
IPv4 and IPv6 (Dual-Stack) |
|
Addressing |
Uses only IPv4 addresses |
Uses both IPv4 and IPv6 addresses |
|
Address Format |
32-bit address (e.g., 192.168.1.1) |
128-bit address (e.g., 2001:db8::1) |
|
Address Space |
Limited address space |
Vastly larger address space |
|
Compatibility |
Compatible with IPv4-only systems |
Compatible with both IPv4 and IPv6 systems |
|
Network Configuration |
Simpler setup, only IPv4 settings |
More complex, requires IPv6 configuration alongside IPv4 |
|
Traffic Handling |
All traffic is IPv4 |
Can handle both IPv4 and IPv6 traffic |
|
Routing |
Routes only IPv4 traffic |
Routes both IPv4 and IPv6 traffic |
|
Firewall Rules |
Configured for IPv4 only |
Must consider rules for both protocols |
|
Transition Mechanisms |
Not applicable |
Supports transition mechanisms like NAT64, 6to4, etc. |
|
Future-Proofing |
Limited future-proofing |
More future-proof as IPv6 adoption increases |
|
End-to-End Connectivity |
Limited by NAT and address shortages |
Better end-to-end connectivity with native addressing |
Alias IP ranges
-
I
n Google Cloud Platform (GCP), Alias IP ranges allow you to assign additional IP addresses to a VM instance's primary network interface. This feature is particularly useful for scenarios where you need to manage multiple IP addresses without creating additional network interfaces.
Network Service Tier
-
T
he Network Service Tier option allows you to choose the level of network performance and pricing that best suits your application's needs.
It has two options:
Premium
Standard – 200 GB/month free in every region
4.12 Compute Engine - Observability: The Observability in Google Cloud Platform (GCP) Compute Engine focuses on monitoring, logging, and gaining insights into your VM instances and overall infrastructure. Observe your instance and application through collection of logs and metrics. When you enable Install Ops Agent for Monitoring and Logging, it adds cost for logging and Monitoring.
4.13 Compute Engine – Security – It protects your virtual machines from various threats, including rootkits and bootkits.
Secure Boot: This feature prevents unauthorized code from running during the boot process by ensuring only signed and trusted code is executed.
Virtual Trusted Platform Module (
vTPM
): Shielded VMs use a virtual TPM to store cryptographic keys and measurements securely, adding an extra layer of protection for sensitive data.
Integrity Monitoring: Shielded VMs verify the integrity of the boot process, ensuring that the VM starts up with a trusted and verified boot image.
Turn on all settings for the most secure configuration.
VM access – It Manage how users connect to the VM. By default, when you connect to a VM using this console or gcloud, your SSH keys are generated automatically. There two options available:
Control VM access through IAM permissions
-
Link VM access to the user's IAM role. Enables OS Login
Block project-wide SSH keys
-
When ticked, project-wide SSH keys cannot access this instance.
You can also add your own keys for VM access through a third-party tool. You cannot use these keys when IAM-based access (using OS Login) is enabled.
4.14 Compute Engine – Management
Deletion protection
– when this option is enabled, a virtual machine
cannot
be deleted.
Reservations
-
Reservations
for Compute Engine instances refer to the allocation of specific resources (like CPU and memory) for your virtual machines (VMs).
Reservation Options –
Automatically use when created:
If you select this option, the VM will automatically use the reserved resources you have set up. This is beneficial if you want to guarantee that your VM has the necessary resources available, especially in scenarios where you expect high usage or need to maintain performance levels.
Don't use:
Choosing this option means that the VM will not use any reserved resources when it is created. Instead, it will draw resources from the general pool of available resources in the region or zone.
This might be suitable if you're flexible about resource availability or are managing workloads that do not require guaranteed capacity. However, it could lead to issues if demand exceeds the available capacity in the general pool.
Automation
-
A startup script is a file that contains commands that run when a virtual machine (VM) instance boots. Compute Engine provides support for running startup scripts on Linux VMs and Windows VMs.
The following table contains links to documentation that describes how to use startup scripts.
|
Startup script task |
Best used for |
Link to procedure |
|
|
Pass a startup script directly |
Scripts up to 256 KB |
||
|
Pass a startup script from a local file |
Scripts up to 256 KB |
||
|
Pass a startup script from Cloud Storage |
Scripts greater than 256 KB |
||
|
Access metadata from a startup script |
Passing different values to each VM that uses the same script |
||
|
Rerun a startup script |
Running a script without restarting the VM |
||
|
View the output from a startup script |
Monitoring the progress of a script |
||
4.15 Instance Template
4.16 Sole Tenant Node
4.17 Machine Image
4.18 TPUs
4.19 Committed Use Discount
4.20 Reservations
4.21 Migrate to Virtual Machine
4.22 Storage:
4.22.1 Disks
4.22.1 Storage Pools
4.22.1 Snapshots
4.22.1 Image
4.22.1 Async replication
4.23 Instance Group
4.23.1 Instance Groups
4.23.2 Health Checks
4.24 VM Manager
4.24.1 Patch
4.24.2 OS Policies
4.25 Bare Metal Solution: You have to plan and make an order to Google to get the physical hardware which is completely dedicated for you.
dedicated physical servers provided to customers without virtualization layers.
bare metal instances allow to run workloads directly on physical hardware
Has high performance
Complete isolated from other users
You can customize your machine as per your need
You can bring your own software license
4.26 Settings
4.3.1 Standard Virtual Machine: In this category, virtual machines has predefined vCPU, memory.
4.3.2 Custom Virtual Machine
4.3.3 Spot Virtual Machine
4.3.4 Pre-emptive Virtual Machine
4.3.5 Sole Tenant Nodes
4.3.6 Bare Matel Solution
|
Machine Family |
Machine Series |
Description |
|
General-Purpose |
Cost-optimized: E2Balanced: N1, N2, N2DScale-out optimized: Tau T2D, Tau T2A |
|
|
Compute-Optimized |
C2, C2D |
These instances offer more CPU power and better performance, with a choice of sizing and processing technologies. |
|
Memory-Optimized |
M1, M2 |
These instances are used when more memory is needed. They provide the most affordable price per GB of memory of all instance types. |
|
GPU-Optimized |
A2 |
This type makes use of GPUs shared among several instances. They are typically very expensive and only used for graphics-intensive workloads requiring GPU processing. |
|
Instance Type |
Processor Types |
Maximum vCPUs |
Maximum Memory |
Local SSD |
Maximum Network Speed (Gbps) |
GPU |
|
E2 |
Intel and AMD EPYC |
32 |
128 GB |
No |
16 |
No |
|
N2 |
Intel Ice Lake and Cascade Lake |
128 |
864 GB |
Yes |
100 |
No |
|
Tau T2D |
AMD EPYC Milan |
60 |
240 GB |
No |
32 |
No |
|
C2 |
Intel Cascade Lake |
60 |
240 GB |
Yes |
100 |
No |
|
C2D |
AMD EPYC Milan |
112 |
224 GB |
Yes |
100 |
No |
|
M1 |
Intel Broadwell |
160 |
4 TB |
Yes |
32 |
No |
|
M2 |
Intel Cascade Lake |
416 |
12 TB |
No |
32 |
No |
|
A2 |
Intel Cascade Lake |
96 |
1.5 TB |
Yes |
100 |
Yes |
Addional Notes on Observability:
Key Components of Observability in GCP Compute Engine:
Stackdriver
Monitoring:
Overview: Now part of Google Cloud Operations Suite,
Stackdriver
Monitoring provides insights into the performance, uptime, and overall health of your VM instances.
Features:
Dashboards: Create custom dashboards to visualize key metrics like CPU usage, memory consumption, disk I/O, and network traffic.
Alerts: Set up alerts based on specific thresholds or conditions to notify you of potential issues before they affect users.
Integration: Integrates with other GCP services and allows for monitoring of both GCP and on-premises resources.
Stackdriver
Logging:
Overview: Part of Google Cloud Operations Suite,
Stackdriver
Logging allows you to collect and
analyze
logs from your VM instances and applications.
Features:
Log Management: Collect logs from various sources, including system logs, application logs, and audit logs.
Log Queries: Use powerful querying capabilities to search through logs and filter by specific criteria.
Retention and Export: Manage log retention policies and export logs to other GCP services like
BigQuery
for further analysis.
Cloud Trace:
Overview: Cloud Trace helps you
analyze
the latency of your applications by tracking the time it takes for requests to travel through various services and components.
Features:
Distributed Tracing: Provides insights into how different parts of your application interact and where bottlenecks may occur.
Performance Optimization: Helps identify slow components or services, enabling you to optimize performance.
Cloud Profiler:
Overview: Cloud Profiler provides continuous profiling of your applications, helping you understand resource consumption and optimize performance.
Features:
CPU and Memory Usage: Visualizes how much CPU and memory your applications use over time.
Code Optimization: Identifies areas in your code that may be consuming excessive resources, helping to optimize performance.
Error Reporting:
Overview: Automatically aggregates and displays the crashes and errors in your application, providing insights into stability issues.
Features:
Real-time Reporting: Captures and reports errors as they occur, allowing for quick diagnosis and resolution.
Integration: Works seamlessly with other GCP services to provide a unified view of application health.
Service Monitoring:
Overview: Monitors the performance and availability of services running on your Compute Engine instances.
Features:
Health Checks: Automatically performs health checks on your services to ensure they are running as expected.
Service Level Objectives (SLOs): Set SLOs to measure the performance and reliability of your services.
