As software engineers, we’re all too familiar with the challenges that come with maintaining and documenting APIs. While REST and tools like Swagger (OpenAPI) have served us well, they often come with their own set of issues — versioning, documentation drift, and the tedious task of manually keeping everything in sync. Today, I want to introduce gRPC, a high-performance RPC (Remote Procedure Call) framework that can streamline these processes, offering a solution that’s both robust and developer-friendly.
In this article, we’ll explore the basics of gRPC and how it can simplify the complexity of microservice architecture through code generation.
Wewill write a simple user management microservice to demonstrate how easy and clear it is to work with gRPC.
What is gRPC?
gRPC is a modern open-source RPC framework developed by Google. It uses Protocol Buffers (protobufs) as its Interface Definition Language (IDL), allowing you to define your service’s interface clearly in a .proto file. This file serves as a single source of truth for your application.
Based on this .proto file, gRPC generates code in the language of your choice (almost all programming languages are supported). You’re then left only with implementing the logic of the methods you defined in the protobuf, for which gRPC has already generated the signatures.
The Advantages of gRPC
- Code Generation: With gRPC, you define your service once in a
.protofile, and gRPC generates the client and server stubs automatically. This eliminates the need for manual API documentation and ensures consistency between your services. - High Performance: gRPC uses HTTP/2, which is faster and more efficient than HTTP/1.1 used by traditional REST APIs. This makes gRPC a great choice for microservices that require low-latency communication.
- Strongly Typed Contracts: gRPC enforces strict contracts between services, which can help catch errors at compile time rather than runtime. This makes your services more reliable and easier to debug.
- Built-in Versioning: Protocol Buffers inherently support versioning, making it easier to evolve your APIs without breaking existing clients.
For your API’s users, the gRPC
.protofile is like a subway map for a traveler. It provides all the information the user needs to know about the service without burdening them with the complex details of the implementation
gRPC lets you write a document in the perfect middle between English and Code
Example: A Simple User Management Microservice
Let’s dive into a practical example. We’ll create a user management microservice called user. Here’s the user.proto file defining the service :
syntax = "proto3";
import "google/protobuf/empty.proto";
import "google/protobuf/field_mask.proto";
package user.v1;
// The UserService provides information and management for users.
service UserService {
// Method that creates a new user.
rpc CreateUser (CreateUserRequest) returns (User);
// Method that gets a user by ID
rpc GetUser (GetUserRequest) returns (User);
// Method that updates an existing user.
rpc UpdateUser (UpdateUserRequest) returns (User);
// Method that deletes a user by ID.
rpc DeleteUser (DeleteUserRequest) returns (google.protobuf.Empty);
}
message GetUserRequest {
string id = 1; // the unique ID of the user
}
message CreateUserRequest {
string name = 1; // Name of the user
string email = 2; // Email of the user
Geolocation location = 3; // optional. Geolocation of the user
}
message UpdateUserRequest {
User user = 1; // The user with updated fields
google.protobuf.FieldMask update_mask = 2; // Fields to be updated
}
message DeleteUserRequest {
string id = 1; // the unique ID of the user
}
// Message to represent a user
message User {
string id = 1; // Unique identifier for the user
string name = 2; // Name of the user
string email = 3; // Email of the user
Geolocation location = 4; // optional. Geolocation of the user
}
// Message to represent geolocation
message Geolocation {
double latitude = 1;
double longitude = 2;
}
As you can see, this API offers four methods: CreateUser, GetUser, UpdateUser and DeleteUser. Each of these functions receives an input and returns an output. The input and output of RPC functions are called messages in gRPC.
Protocol buffer data is structured as messages, where each message is a small logical record of information containing a series of name-value pairs called fields. The messages you define in your protobuf describe what information you send to the service methods and what you get in return.
For example, CreateUserreceives a CreateUserRequestmessage, which is defined as the name, email, and Geolocation of the user we want to create. Geolocation is itself a message defined as latitude and longitude. In return, we receive the User message.
You can think of a gRPC message as an object containing information that will then be binary serialized for very fast data transfer.
The UpdateUserRequest method is also interesting. It receives as input a User message and a FieldMask message defining the fields to select in the User given for the update. This is the standard way of updating entities in gRPC according to the Google convention.
Another advantage of gRPC is that if you stick to its convention, one could even guess the behavior of your API without needing to read its .proto file
Generating Code from the .proto File
With gRPC, we can generate the Python (or any other language) code that implements the server and client stubs for this service. Here’s how you can do it:
# update pip
python -m pip install --upgrade pip
# install grpcio-tools package
pip install grpcio-tools
# generate python code based on the user.proto file we defined
python -m grpc_tools.protoc -I. --python_out=./stubs/ --grpc_python_out=./stubs/ user.proto
After running this command, you’ll have the follwing 2 Python files in the stubs/ directory:
This is the first fileuser_pb2.py
# -*- coding: utf-8 -*-
# Generated by the protocol buffer compiler. DO NOT EDIT!
# NO CHECKED-IN PROTOBUF GENCODE
# source: user.proto
# Protobuf Python Version: 5.27.2
"""Generated protocol buffer code."""
from google.protobuf import descriptor as _descriptor
from google.protobuf import descriptor_pool as _descriptor_pool
from google.protobuf import runtime_version as _runtime_version
from google.protobuf import symbol_database as _symbol_database
from google.protobuf.internal import builder as _builder
_runtime_version.ValidateProtobufRuntimeVersion(
_runtime_version.Domain.PUBLIC,
5,
27,
2,
'',
'user.proto'
)
# @@protoc_insertion_point(imports)
_sym_db = _symbol_database.Default()
from google.protobuf import empty_pb2 as google_dot_protobuf_dot_empty__pb2
from google.protobuf import field_mask_pb2 as google_dot_protobuf_dot_field__mask__pb2
DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\nuser.proto\x12\x07user.v1\x1a\x1bgoogle/protobuf/empty.proto\x1a google/protobuf/field_mask.proto\"\x1c\n\x0eGetUserRequest\x12\n\n\x02id\x18\x01 \x01(\t\"X\n\x11\x43reateUserRequest\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\r\n\x05\x65mail\x18\x02 \x01(\t\x12&\n\x08location\x18\x03 \x01(\x0b\x32\x14.user.v1.Geolocation\"a\n\x11UpdateUserRequest\x12\x1b\n\x04user\x18\x01 \x01(\x0b\x32\r.user.v1.User\x12/\n\x0bupdate_mask\x18\x02 \x01(\x0b\x32\x1a.google.protobuf.FieldMask\"\x1f\n\x11\x44\x65leteUserRequest\x12\n\n\x02id\x18\x01 \x01(\t\"W\n\x04User\x12\n\n\x02id\x18\x01 \x01(\t\x12\x0c\n\x04name\x18\x02 \x01(\t\x12\r\n\x05\x65mail\x18\x03 \x01(\t\x12&\n\x08location\x18\x04 \x01(\x0b\x32\x14.user.v1.Geolocation\"2\n\x0bGeolocation\x12\x10\n\x08latitude\x18\x01 \x01(\x01\x12\x11\n\tlongitude\x18\x02 \x01(\x01\x32\xf4\x01\n\x0bUserService\x12\x37\n\nCreateUser\x12\x1a.user.v1.CreateUserRequest\x1a\r.user.v1.User\x12\x31\n\x07GetUser\x12\x17.user.v1.GetUserRequest\x1a\r.user.v1.User\x12\x37\n\nUpdateUser\x12\x1a.user.v1.UpdateUserRequest\x1a\r.user.v1.User\x12@\n\nDeleteUser\x12\x1a.user.v1.DeleteUserRequest\x1a\x16.google.protobuf.Emptyb\x06proto3')
_globals = globals()
_builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, _globals)
_builder.BuildTopDescriptorsAndMessages(DESCRIPTOR, 'user_pb2', _globals)
if not _descriptor._USE_C_DESCRIPTORS:
DESCRIPTOR._loaded_options = None
_globals['_GETUSERREQUEST']._serialized_start=86
_globals['_GETUSERREQUEST']._serialized_end=114
_globals['_CREATEUSERREQUEST']._serialized_start=116
_globals['_CREATEUSERREQUEST']._serialized_end=204
_globals['_UPDATEUSERREQUEST']._serialized_start=206
_globals['_UPDATEUSERREQUEST']._serialized_end=303
_globals['_DELETEUSERREQUEST']._serialized_start=305
_globals['_DELETEUSERREQUEST']._serialized_end=336
_globals['_USER']._serialized_start=338
_globals['_USER']._serialized_end=425
_globals['_GEOLOCATION']._serialized_start=427
_globals['_GEOLOCATION']._serialized_end=477
_globals['_USERSERVICE']._serialized_start=480
_globals['_USERSERVICE']._serialized_end=724
# @@protoc_insertion_point(module_scope)
… and this is the second file user_pb2_grpc.py :
# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT!
"""Client and server classes corresponding to protobuf-defined services."""
import grpc
import warnings
from google.protobuf import empty_pb2 as google_dot_protobuf_dot_empty__pb2
import user_pb2 as user__pb2
GRPC_GENERATED_VERSION = '1.66.1'
GRPC_VERSION = grpc.__version__
_version_not_supported = False
try:
from grpc._utilities import first_version_is_lower
_version_not_supported = first_version_is_lower(GRPC_VERSION, GRPC_GENERATED_VERSION)
except ImportError:
_version_not_supported = True
if _version_not_supported:
raise RuntimeError(
f'The grpc package installed is at version {GRPC_VERSION},'
+ f' but the generated code in user_pb2_grpc.py depends on'
+ f' grpcio>={GRPC_GENERATED_VERSION}.'
+ f' Please upgrade your grpc module to grpcio>={GRPC_GENERATED_VERSION}'
+ f' or downgrade your generated code using grpcio-tools<={GRPC_VERSION}.'
)
class UserServiceStub(object):
"""The UserService provides information and management for users.
"""
def __init__(self, channel):
"""Constructor.
Args:
channel: A grpc.Channel.
"""
self.CreateUser = channel.unary_unary(
'/user.v1.UserService/CreateUser',
request_serializer=user__pb2.CreateUserRequest.SerializeToString,
response_deserializer=user__pb2.User.FromString,
_registered_method=True)
self.GetUser = channel.unary_unary(
'/user.v1.UserService/GetUser',
request_serializer=user__pb2.GetUserRequest.SerializeToString,
response_deserializer=user__pb2.User.FromString,
_registered_method=True)
self.UpdateUser = channel.unary_unary(
'/user.v1.UserService/UpdateUser',
request_serializer=user__pb2.UpdateUserRequest.SerializeToString,
response_deserializer=user__pb2.User.FromString,
_registered_method=True)
self.DeleteUser = channel.unary_unary(
'/user.v1.UserService/DeleteUser',
request_serializer=user__pb2.DeleteUserRequest.SerializeToString,
response_deserializer=google_dot_protobuf_dot_empty__pb2.Empty.FromString,
_registered_method=True)
class UserServiceServicer(object):
"""The UserService provides information and management for users.
"""
def CreateUser(self, request, context):
"""Creates a new user.
"""
context.set_code(grpc.StatusCode.UNIMPLEMENTED)
context.set_details('Method not implemented!')
raise NotImplementedError('Method not implemented!')
def GetUser(self, request, context):
"""Get a user by ID
"""
context.set_code(grpc.StatusCode.UNIMPLEMENTED)
context.set_details('Method not implemented!')
raise NotImplementedError('Method not implemented!')
def UpdateUser(self, request, context):
"""Updates an existing user.
"""
context.set_code(grpc.StatusCode.UNIMPLEMENTED)
context.set_details('Method not implemented!')
raise NotImplementedError('Method not implemented!')
def DeleteUser(self, request, context):
"""Deletes a user by ID.
"""
context.set_code(grpc.StatusCode.UNIMPLEMENTED)
context.set_details('Method not implemented!')
raise NotImplementedError('Method not implemented!')
def add_UserServiceServicer_to_server(servicer, server):
rpc_method_handlers = {
'CreateUser': grpc.unary_unary_rpc_method_handler(
servicer.CreateUser,
request_deserializer=user__pb2.CreateUserRequest.FromString,
response_serializer=user__pb2.User.SerializeToString,
),
'GetUser': grpc.unary_unary_rpc_method_handler(
servicer.GetUser,
request_deserializer=user__pb2.GetUserRequest.FromString,
response_serializer=user__pb2.User.SerializeToString,
),
'UpdateUser': grpc.unary_unary_rpc_method_handler(
servicer.UpdateUser,
request_deserializer=user__pb2.UpdateUserRequest.FromString,
response_serializer=user__pb2.User.SerializeToString,
),
'DeleteUser': grpc.unary_unary_rpc_method_handler(
servicer.DeleteUser,
request_deserializer=user__pb2.DeleteUserRequest.FromString,
response_serializer=google_dot_protobuf_dot_empty__pb2.Empty.SerializeToString,
),
}
generic_handler = grpc.method_handlers_generic_handler(
'user.v1.UserService', rpc_method_handlers)
server.add_generic_rpc_handlers((generic_handler,))
server.add_registered_method_handlers('user.v1.UserService', rpc_method_handlers)
# This class is part of an EXPERIMENTAL API.
class UserService(object):
"""The UserService provides information and management for users.
"""
@staticmethod
def CreateUser(request,
target,
options=(),
channel_credentials=None,
call_credentials=None,
insecure=False,
compression=None,
wait_for_ready=None,
timeout=None,
metadata=None):
return grpc.experimental.unary_unary(
request,
target,
'/user.v1.UserService/CreateUser',
user__pb2.CreateUserRequest.SerializeToString,
user__pb2.User.FromString,
options,
channel_credentials,
insecure,
call_credentials,
compression,
wait_for_ready,
timeout,
metadata,
_registered_method=True)
@staticmethod
def GetUser(request,
target,
options=(),
channel_credentials=None,
call_credentials=None,
insecure=False,
compression=None,
wait_for_ready=None,
timeout=None,
metadata=None):
return grpc.experimental.unary_unary(
request,
target,
'/user.v1.UserService/GetUser',
user__pb2.GetUserRequest.SerializeToString,
user__pb2.User.FromString,
options,
channel_credentials,
insecure,
call_credentials,
compression,
wait_for_ready,
timeout,
metadata,
_registered_method=True)
@staticmethod
def UpdateUser(request,
target,
options=(),
channel_credentials=None,
call_credentials=None,
insecure=False,
compression=None,
wait_for_ready=None,
timeout=None,
metadata=None):
return grpc.experimental.unary_unary(
request,
target,
'/user.v1.UserService/UpdateUser',
user__pb2.UpdateUserRequest.SerializeToString,
user__pb2.User.FromString,
options,
channel_credentials,
insecure,
call_credentials,
compression,
wait_for_ready,
timeout,
metadata,
_registered_method=True)
@staticmethod
def DeleteUser(request,
target,
options=(),
channel_credentials=None,
call_credentials=None,
insecure=False,
compression=None,
wait_for_ready=None,
timeout=None,
metadata=None):
return grpc.experimental.unary_unary(
request,
target,
'/user.v1.UserService/DeleteUser',
user__pb2.DeleteUserRequest.SerializeToString,
google_dot_protobuf_dot_empty__pb2.Empty.FromString,
options,
channel_credentials,
insecure,
call_credentials,
compression,
wait_for_ready,
timeout,
metadata,
_registered_method=True)
Unreadable to you ? No worry ! I will show you how to make use of it very easily. We will import it and inherit the methods. Let’s define a server.py that will serve this microservice using the above code generated.
import grpc
from concurrent import futures
from grpc_reflection.v1alpha import reflection
import user_pb2
import user_pb2_grpc
from google.protobuf import empty_pb2, field_mask_pb2
from dao.user_dao import UserDAOMongoDB # dao class to interact with MongoDB
SERVICE_NAME = 'UserService'
SERVICE_PORT = '50051'
class UserServiceServicer(user_pb2_grpc.UserServiceServicer): # Here I inherit from the generated interface user_pb2_grpc.UserServiceServicer
def __init__(self):
self.dao = UserDAOMongoDB()
# Below I override all the already methods defined in the
# user_pb2_grpc.UserServiceServicer and generated directly
# from my .proto file. All is left for me is to write the code they
# should execute and that's all.
def CreateUser(self, request, context):
# Save the user in the mongoDB
user = self.dao.create_user(request.name, request.email, request.location)
return user
def GetUser(self, request, context):
# Retrieve the user from the mongoDB
user = self.dao.get_user(request.id)
if not user:
context.set_code(grpc.StatusCode.NOT_FOUND)
context.set_details('User not found')
return user_pb2.User()
return user
def UpdateUser(self, request, context):
# Update the user in the mongoDB
updated_user = self.dao.update_user(request.user, request.update_mask)
if not updated_user:
context.set_code(grpc.StatusCode.NOT_FOUND)
context.set_details('User not found')
return user_pb2.User()
return updated_user
def DeleteUser(self, request, context):
# Delete a user from mongoDB
self.dao.delete_user(request.id)
return empty_pb2.Empty()
def serve():
server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
user_pb2_grpc.add_UserServiceServicer_to_server(UserServiceServicer(), server)
SERVICE_NAMES = (
user_pb2.DESCRIPTOR.services_by_name[SERVICE_NAME].full_name,
reflection.SERVICE_NAME,
)
reflection.enable_server_reflection(SERVICE_NAMES, server)
server.add_insecure_port(f'[::]:{SERVICE_PORT}')
server.start()
server.wait_for_termination()
if __name__ == '__main__':
print(f"Serving {SERVICE_NAME} on gRPC on port {SERVICE_PORT} !!!")
serve()
Run python server.py and that’s all it takes to have your microservice running:
Serving UserService on gRPC on port 50051 !!!
Communicate with your gRPC App
Once you’ve generated the gRPC code from your .proto file, you can easily create either a server app (like we just did) or a client app. The beauty of gRPC is that it handles all the complexities of serialization, deserialization, and networking, allowing you to focus on the business logic of your application.
While creating a client in Python is straightforward, I won’t dive into that here. Instead, I want to introduce you to a powerful tool called grpcurl that you can use to interact with your gRPC services directly from the command line.
grpcurl is a command-line tool that works similarly to curl, but for gRPC services. It allows you to invoke gRPC methods and inspect gRPC endpoints with ease, making it an essential tool for testing and debugging gRPC services. Check out their GitHub project for installation procedure.
Exploring Your gRPC Service with grpcurl
Before interacting with your service, it’s often helpful to explore what your service offers. Here’s how you can use grpcurl to do that.
Describe the Service
You can describe your entire gRPC service to see what methods are available:
$ grpcurl -plaintext localhost:50051 describe user.v1.UserService
user.v1.UserService is a service:
service UserService {
rpc CreateUser ( .user.v1.CreateUserRequest ) returns ( .user.v1.User );
rpc GetUser ( .user.v1.GetUserRequest ) returns ( .user.v1.User );
rpc UpdateUser ( .user.v1.UpdateUserRequest ) returns ( .user.v1.User );
rpc DeleteUser ( .user.v1.DeleteUserRequest ) returns ( .google.protobuf.Empty );
}
Describe a Specific Method
$ grpcurl -plaintext localhost:50051 describe user.v1.UserService.CreateUser
user.v1.UserService.CreateUser is a method:
rpc CreateUser ( .user.v1.CreateUserRequest ) returns ( .user.v1.User );
Interacting with the UserService
- Create a user
$ grpcurl -d '{"name": "John Doe", "email": "john.doe@example.com"}' \
-plaintext localhost:50051 user.v1.UserService/CreateUser
{
"id": "12345",
"name": "John Doe",
"email": "john.doe@example.com"
}
2. Get a user by ID
$ grpcurl -d '{"id": "12345"}' \
-plaintext localhost:50051 user.v1.UserService/GetUser
{
"id": "12345",
"name": "John Doe",
"email": "john.doe@example.com"
}
3. Update a user
$ grpcurl -d '{
"user": {
"id": "12345",
"name": "Johnathan Doe"
},
"update_mask": {
"paths": ["name"]
}
}' -plaintext localhost:50051 user.v1.UserService/UpdateUser
{
"id": "12345",
"name": "Johnathan Doe",
"email": "john.doe@example.com"
}
4. Delete a user
grpcurl -d '{"id": "12345"}' \
-plaintext localhost:50051 user.v1.UserService/DeleteUser
{}
Conclusion
In the age of generative AI, it’s crucial for software professionals to adopt frameworks that ensure clarity and control. gRPC stands out by combining human-written Protocol Buffer definitions with AI-driven implementation, creating a powerful duo that enhances code quality and team performance. Without clear protobuf definitions, AI implementations can become unpredictable and harder to manage. By leveraging gRPC, you ensure your APIs are well-defined and your AI integrations remain reliable and efficient. Embrace gRPC to boost your development workflow and achieve greater consistency in your microservices architecture.