<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.10.0">Jekyll</generator><link href="https://lpsandaruwan.github.io/feed.xml" rel="self" type="application/atom+xml" /><link href="https://lpsandaruwan.github.io/" rel="alternate" type="text/html" /><updated>2025-12-16T06:40:13+00:00</updated><id>https://lpsandaruwan.github.io/feed.xml</id><title type="html">Lahiru’s dev journal</title><subtitle>Building software since 2016. Now focused on systems that think and scale, mostly about AI agents and serverless architecture. Computer science graduate who loves Linux, lives in the terminal, and believes open source makes everything better.</subtitle><author><name>Lahiru Sandaruwan Pathirage</name><email>hello@lpsandaruwan.dev</email></author><entry><title type="html">I’ve Noticed AI Tools Generate Terrible REST APIs - Here’s How to Fix It</title><link href="https://lpsandaruwan.github.io/posts/ive-noticed-ai-tools-generate-terrible-rest-apis/" rel="alternate" type="text/html" title="I’ve Noticed AI Tools Generate Terrible REST APIs - Here’s How to Fix It" /><published>2025-07-10T00:00:00+00:00</published><updated>2025-07-10T00:00:00+00:00</updated><id>https://lpsandaruwan.github.io/posts/ive-noticed-ai-tools-generate-terrible-rest-apis</id><content type="html" xml:base="https://lpsandaruwan.github.io/posts/ive-noticed-ai-tools-generate-terrible-rest-apis/"><![CDATA[<p>Here’s something that bugs me: AI tools like ChatGPT, Claude, Gemini, or whatever AI you’re using are great at cranking out code fast,
but they often generate REST APIs that look like they were designed by someone who never had to maintain them.</p>

<p>Over the past few years working with different teams, I’ve noticed that about most of AI generated APIs have naming conventions that make me question everything.</p>

<p><img src="/assets/images/content/posts/rest/mrCroca.png" alt="REST API design illustration" style="max-width: 800px; height: auto; display: block; margin: 20px auto;" /></p>

<p>Why does this happen? AI models learn from existing code on the internet, and unfortunately, there’s a lot of bad API design out there. When you ask an AI to “create a book API,” it might give you something like this:</p>

<p><code class="language-plaintext highlighter-rouge">GET https://example.com/api/books/get-one/{book-id}</code></p>

<p><code class="language-plaintext highlighter-rouge">GET https://example.com/api/books/get-all</code></p>

<p><code class="language-plaintext highlighter-rouge">GET https://example.com/api/books/filter-by-author-id/authorId=1&amp;autherId=2&amp;autherId=3...</code></p>

<p><code class="language-plaintext highlighter-rouge">POST https://example.com/api/create-book</code></p>

<p><code class="language-plaintext highlighter-rouge">PUT https://example.com/api/update-book</code></p>

<p>If your APIs look like this, don’t take it personally. This post isn’t meant to shame anyone. The problem is that AI tools often copy patterns from Stack Overflow answers, old tutorials, and legacy codebases where people were just trying to get something working quickly.</p>

<p>But here’s the thing: APIs are interfaces that other developers (including future you) will have to work with. A poorly designed API can waste hours of development time, create confusion in documentation, and make your codebase harder to maintain.</p>

<p>Let me show you why these examples are problematic and how to fix them.</p>

<h2 id="too-bored-to-read-copy-this-prompt">Too Bored to Read? Copy This Prompt</h2>

<p>If you’re too bored to read this entire post, just copy and paste this comprehensive prompt into your AI coding assistant so it follows these guidelines when generating REST APIs. Instructions for each tool are included below:</p>

<div style="border: 1px solid var(--border-color, #444); border-radius: 5px; margin: 10px 0;">
  <div style="background: var(--code-bg, #2d2d2d); padding: 10px; border-bottom: 1px solid var(--border-color, #444); cursor: pointer; color: var(--text-color, #fff);" onclick="this.nextElementSibling.style.display = this.nextElementSibling.style.display === 'none' ? 'block' : 'none'">
    📋 Click to expand comprehensive AI prompt for REST API guidelines
  </div>
  <div style="display: none; position: relative;">
    <button onclick="navigator.clipboard.writeText(this.nextElementSibling.textContent)" style="position: absolute; top: 10px; right: 10px; background: var(--accent-color, #007cba); color: white; border: none; padding: 5px 10px; border-radius: 3px; cursor: pointer; font-size: 12px;">Copy</button>
    <pre style="background: var(--code-bg, #1e1e1e); color: var(--code-color, #fff); padding: 15px; margin: 0; overflow-x: auto; white-space: pre-wrap;"><code>You are a professional AI coding assistant. Follow these REST API design rules religiously when generating any REST API related code, endpoints, or architecture.

## MANDATORY REST API RULES

### URL DESIGN
- Use nouns, NEVER verbs in URLs
- Collections MUST be plural: /api/v1/books not /api/v1/book
- Always version APIs: /api/v1/, /api/v2/
- Structure: {protocol}://{host}/api/{version}/{collection}/{id}?{query_params}

### HTTP METHODS
- GET: Read operations only (safe, idempotent, no side effects)
- POST: Create resources or complex operations
- PUT: Replace entire resource (send all fields)
- PATCH: Update partial resource (send only changed fields)
- DELETE: Remove resource

### RESPONSE STANDARDS
- Always include proper HTTP status codes (200, 201, 400, 401, 403, 404, 500)
- Always include pagination for collections: ?page=1&amp;limit=20
- Use consistent naming convention (pick snake_case OR camelCase, never mix)
- Structure error responses with field-level details

### CORRECT PATTERNS
GET    /api/v1/books                    # List books
GET    /api/v1/books/123               # Get specific book
POST   /api/v1/books                   # Create book
PUT    /api/v1/books/123               # Update entire book
PATCH  /api/v1/books/123               # Update book partially
DELETE /api/v1/books/123               # Delete book

GET    /api/v1/authors/456/books       # Author's books (hierarchical)
POST   /api/v1/authors/456/books       # Create book for author

GET    /api/v1/books?author=tolkien&amp;genre=fantasy    # Simple filtering
POST   /api/v1/books/search            # Complex filtering (use POST)

### FORBIDDEN PATTERNS - NEVER GENERATE THESE
❌ GET /api/books/get-one/123
❌ GET /api/books/get-all
❌ POST /api/create-book
❌ PUT /api/update-book
❌ GET /api/getBookById/123
❌ POST /api/createBook
❌ DELETE /api/removeBook/123

### REQUIRED RESPONSE FORMATS

Success Response:
{
  "data": { "id": 123, "title": "The Hobbit" }
}

Collection with Pagination:
{
  "data": [],
  "pagination": {
    "page": 1,
    "limit": 20,
    "total": 150,
    "totalPages": 8
  }
}

Error Response:
{
  "error": {
    "code": "VALIDATION_ERROR",
    "message": "Invalid request data",
    "details": [
      { "field": "price", "message": "Must be positive number" }
    ]
  }
}

### CHECKLIST - VERIFY BEFORE GENERATING
- [ ] URLs use nouns, not verbs
- [ ] Collections are plural
- [ ] Consistent naming throughout
- [ ] Proper HTTP methods
- [ ] API versioning (/api/v1/)
- [ ] Pagination for collections
- [ ] Meaningful status codes
- [ ] Structured error responses

### PROCESS
1. Ask about entities and relationships first
2. Design around resources, not actions
3. Use proper HTTP methods and status codes
4. Include pagination for collection endpoints
5. Provide structured error handling
6. Follow consistent naming conventions
7. Include API versioning

Apply these rules to ALL REST API code generation. These rules are non-negotiable.</code></pre>
  </div>
</div>

<h4 id="how-to-add-these-rules-to-your-ai-tools">How to Add These Rules to Your AI Tools</h4>

<p><strong>For Claude Code (.claude.md):</strong>
Create a <code class="language-plaintext highlighter-rouge">.claude.md</code> file in your project root and paste the rules above.</p>

<p><strong>For Cline (.cline/rules.md):</strong>
Create <code class="language-plaintext highlighter-rouge">.cline/rules.md</code> in your project and paste the rules.</p>

<p><strong>For Cursor (.cursorrules):</strong>
Create a <code class="language-plaintext highlighter-rouge">.cursorrules</code> file in your project root and paste the rules.</p>

<p><strong>For GitHub Copilot:</strong>
Add the rules to your project’s README or create a <code class="language-plaintext highlighter-rouge">copilot-instructions.md</code> file.</p>

<p><strong>For ChatGPT:</strong>
Go to Settings → Personalization → Custom Instructions and paste the rules.</p>

<h2 id="intro">Intro</h2>

<p>I am not going to explain what a REST API is. That you can search in the web, or ask your AI assistant; “Hey ChatGPT. I a noob to REST API. Explain REST API to me like I’m a five year old.”
Main purpose of the REST API is to build a standard or a blueprint where your client and server comes to an agreement to manipulate your resources of business needs.
To make is easier it comes with below options,</p>

<h3 id="uri">URI</h3>
<p>Where is the resource? Ideally it goes like,</p>

<p><code class="language-plaintext highlighter-rouge">{Protocol}://{Host address}/{path and path variables mix}?{query parameter key}={query parameter value}</code></p>

<p>example: <code class="language-plaintext highlighter-rouge">https://example.com/api/v1/vendors/{vendorId}/books?publishedDate=2021-10-10&amp;page=1&amp;limit=100</code></p>

<h3 id="method">Method</h3>

<p>Which type of resource action it is.
There is plenty of methods to use. GET, POST, PUT, …</p>

<h3 id="protocol-and-host-address">Protocol and Host address</h3>

<p>The server’s address or mapped domain including the protocol(http or https)</p>

<h3 id="path-variables">Path variables</h3>

<p>Variables which are included in the URI path. Mainly path variables are for representing a main entity or a hierarchical resources.
i.e. Vendors have books &lt;-&gt; Books are owned by vendors
<code class="language-plaintext highlighter-rouge">https://example.com/api/v1/vendors/{veendorId}/books/{bookId}</code></p>

<h3 id="query-parameters">Query parameters</h3>

<p>Mostly using for extra options and filtering resources.
i.e. filter books by published date
<code class="language-plaintext highlighter-rouge">https://example.com/api/v1/books/publishedDate=2021-05-12</code></p>

<h2 id="designing">Designing</h2>

<p>Designing is the crucial part of every system. At least you need to have a virtual or a mind map. So let’s talk about it.
Now we have the idea about the parts of a RESTful API. If you get a problem to solve or to design a system with REST APIs,
let’s see how we can put good conventions and good practices into the design.</p>

<p>It is best to always design rest apis around entities and their relationships. It doesn’t need to have a visible model or a database,
the entities can be virtual too.</p>

<p>Let’s take a look at examples.</p>

<h3 id="example-1-book-store-system">Example 1: Book Store System</h3>

<p>Consider a simple book store with these entities:</p>
<ul>
  <li><strong>Authors</strong> (id, name, email)</li>
  <li><strong>Books</strong> (id, title, isbn, price, author_id)</li>
  <li><strong>Customers</strong> (id, name, email)</li>
  <li><strong>Orders</strong> (id, customer_id, order_date, total)</li>
  <li><strong>Order Items</strong> (order_id, book_id, quantity, price)</li>
</ul>

<p>Here’s how you’d design clean REST APIs for this system:</p>

<p><strong>Books Management:</strong></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>GET    /api/v1/books                    # List all books
GET    /api/v1/books/{bookId}           # Get specific book
POST   /api/v1/books                    # Create new book
PUT    /api/v1/books/{bookId}           # Update entire book
PATCH  /api/v1/books/{bookId}           # Update book partially
DELETE /api/v1/books/{bookId}           # Delete book

GET    /api/v1/books?author={authorId}  # Filter books by author
GET    /api/v1/books?priceMin=10&amp;priceMax=50  # Price range filter
</code></pre></div></div>

<p><strong>Author’s Books (Hierarchical relationship):</strong></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>GET    /api/v1/authors/{authorId}/books # Get all books by author
POST   /api/v1/authors/{authorId}/books # Create book for author
</code></pre></div></div>

<p><strong>Customer Orders:</strong></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>GET    /api/v1/customers/{customerId}/orders     # Customer's orders
POST   /api/v1/customers/{customerId}/orders     # Place new order
GET    /api/v1/orders/{orderId}/items            # Order items
</code></pre></div></div>

<h3 id="example-2-virtual-models---book-recommendation-engine">Example 2: Virtual Models - Book Recommendation Engine</h3>

<p>Sometimes you need APIs for actions that don’t directly map to database tables. Here’s a third party integration example where your book store needs to integrate with an external recommendation service.</p>

<p><strong>Virtual entities we’re working with:</strong></p>
<ul>
  <li><strong>Recommendations</strong> (not stored, generated on demand)</li>
  <li><strong>User Preferences</strong> (aggregated data)</li>
  <li><strong>Search Results</strong> (temporary, filtered data)</li>
</ul>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>GET    /api/v1/users/{userId}/recommendations           # Get recommendations for user
GET    /api/v1/recommendations?type=trending            # Get trending recommendations
POST   /api/v1/recommendations                          # Send user interaction feedback

POST   /api/v1/books                                    # Advanced book search with complex filters
GET    /api/v1/suggestions?q={query}                    # Search suggestions

POST   /api/v1/events                                   # Track user events
GET    /api/v1/users/{userId}/preferences               # Get user preference summary
</code></pre></div></div>

<p>Notice how these endpoints focus on <strong>actions and aggregated data</strong> rather than simple CRUD operations. The recommendation engine doesn’t “store” recommendations - it generates them. But we still follow REST principles by treating them as resources.</p>

<h2 id="conventions-that-actually-matter">Conventions That Actually Matter</h2>

<p>Now here’s the real stuff that separates good APIs from the garbage ones floating around. I’ve been debugging APIs for years, and trust me, following these conventions will save you from a lot of headaches.</p>

<h3 id="1-use-nouns-not-verbs">1. Use Nouns, Not Verbs</h3>

<p>Your URL should describe <strong>what</strong> you’re working with, not <strong>what</strong> you’re doing to it. The HTTP method already tells us the action.</p>

<p><strong>Wrong:</strong></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>POST /api/v1/createBook
GET  /api/v1/getBookById/123
DELETE /api/v1/removeBook/123
</code></pre></div></div>

<p><strong>Right:</strong></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>POST   /api/v1/books
GET    /api/v1/books/123
DELETE /api/v1/books/123
</code></pre></div></div>

<h3 id="2-plural-nouns-for-collections">2. Plural Nouns for Collections</h3>

<p>Collections should always be plural. This keeps your API consistent whether you’re dealing with one item or many.</p>

<p><strong>Wrong:</strong></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>GET /api/v1/book        # Are we getting one book or all books?
GET /api/v1/book/123    # Confusing structure
</code></pre></div></div>

<p><strong>Right:</strong></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>GET /api/v1/books       # Obviously a collection
GET /api/v1/books/123   # Obviously one item from the collection
</code></pre></div></div>

<h3 id="3-http-methods-have-meaning---use-them">3. HTTP Methods Have Meaning - Use Them</h3>

<p>Each HTTP method has a specific purpose. Don’t just use GET and POST for everything.</p>

<ul>
  <li><strong>GET</strong>: Read data, should be safe and idempotent (no side effects)</li>
  <li><strong>POST</strong>: Create new resources or complex operations</li>
  <li><strong>PUT</strong>: Replace entire resource (send all fields)</li>
  <li><strong>PATCH</strong>: Update partial resource (send only changed fields)</li>
  <li><strong>DELETE</strong>: Remove resource</li>
</ul>

<h3 id="4-when-query-parameters-get-messy-use-post">4. When Query Parameters Get Messy, Use POST</h3>

<p>Sometimes your filters become so complex that query parameters look like a mess. That’s when you switch to POST with a request body.</p>

<p><strong>Query parameter nightmare:</strong></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>GET /api/v1/books?author=1&amp;author=2&amp;author=3&amp;genre=fiction&amp;genre=mystery&amp;price_min=10&amp;price_max=50&amp;published_after=2020-01-01&amp;in_stock=true&amp;format=paperback&amp;format=ebook
</code></pre></div></div>

<p><strong>Clean POST approach:</strong></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>POST /api/v1/books/search
{
  "authors": [1, 2, 3],
  "genres": ["fiction", "mystery"],
  "priceRange": {"min": 10, "max": 50},
  "publishedAfter": "2020-01-01",
  "inStock": true,
  "formats": ["paperback", "ebook"]
}
</code></pre></div></div>

<h3 id="5-version-your-apis">5. Version Your APIs</h3>

<p>Always version your APIs from day one. You’ll thank yourself later.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/api/v1/books    # Good
/api/v2/books    # When you need breaking changes
</code></pre></div></div>

<h3 id="6-naming-conventions-matter">6. Naming Conventions Matter</h3>

<p>Pick a naming style and stick with it throughout your entire API. Here are real scenarios where this becomes important:</p>

<p><strong>Scenario 1: E-commerce API with snake_case</strong></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>GET /api/v1/products?created_at=2024-01-01&amp;price_min=50&amp;is_available=true
POST /api/v1/users/123/shipping_addresses
{
  "street_address": "123 Main St",
  "postal_code": "12345",
  "is_default": true
}
</code></pre></div></div>

<p><strong>Scenario 2: Same API with camelCase</strong></p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>GET /api/v1/products?createdAt=2024-01-01&amp;priceMin=50&amp;isAvailable=true
POST /api/v1/users/123/shippingAddresses
{
  "streetAddress": "123 Main St",
  "postalCode": "12345",
  "isDefault": true
}
</code></pre></div></div>

<p>Both work fine, but mixing them creates confusion:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>GET /api/v1/books?created_at=2024-01-01&amp;publishedDate=2024-01-01   # Don't do this!
</code></pre></div></div>

<h3 id="7-return-meaningful-http-status-codes">7. Return Meaningful HTTP Status Codes</h3>

<p>Your status codes should tell the story of what happened:</p>

<ul>
  <li><strong>200 OK</strong>: Everything worked</li>
  <li><strong>201 Created</strong>: New resource created successfully</li>
  <li><strong>400 Bad Request</strong>: Client sent invalid data</li>
  <li><strong>401 Unauthorized</strong>: Authentication required</li>
  <li><strong>403 Forbidden</strong>: Authenticated but not allowed</li>
  <li><strong>404 Not Found</strong>: Resource doesn’t exist</li>
  <li><strong>500 Internal Server Error</strong>: Server screwed up</li>
</ul>

<p>Don’t just return 200 for everything and put the real status in the response body. That’s lazy.</p>

<h2 id="real-world-lessons-ive-learned">Real World Lessons I’ve Learned</h2>

<p>After working with APIs in production for years, here are some things that books won’t tell you:</p>

<h3 id="pagination-is-not-optional">Pagination is Not Optional</h3>

<p>If your API returns lists, it needs pagination. I don’t care if you think your table will only have 50 records forever. Trust me, it won’t.</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>GET /api/v1/books?page=1&amp;limit=20
GET /api/v1/books?offset=0&amp;limit=20     # Alternative approach
</code></pre></div></div>

<p>Return metadata about the pagination:</p>
<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
  </span><span class="nl">"data"</span><span class="p">:</span><span class="w"> </span><span class="p">[],</span><span class="w">
  </span><span class="nl">"pagination"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
    </span><span class="nl">"page"</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w">
    </span><span class="nl">"limit"</span><span class="p">:</span><span class="w"> </span><span class="mi">20</span><span class="p">,</span><span class="w">
    </span><span class="nl">"total"</span><span class="p">:</span><span class="w"> </span><span class="mi">150</span><span class="p">,</span><span class="w">
    </span><span class="nl">"total_pages"</span><span class="p">:</span><span class="w"> </span><span class="mi">8</span><span class="w">
  </span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>

<h3 id="filter-parameters-should-be-intuitive">Filter Parameters Should Be Intuitive</h3>

<p>Don’t make people guess how to filter your data. Make it obvious:</p>

<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>GET /api/v1/books?author=tolkien&amp;genre=fantasy&amp;available=true
GET /api/v1/books?priceMin=10&amp;priceMax=50
GET /api/v1/books?publishedAfter=2020-01-01
</code></pre></div></div>

<h3 id="error-responses-need-structure">Error Responses Need Structure</h3>

<p>When things go wrong (and they will), return helpful error messages:</p>

<div class="language-json highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span><span class="w">
  </span><span class="nl">"error"</span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
    </span><span class="nl">"code"</span><span class="p">:</span><span class="w"> </span><span class="s2">"VALIDATION_ERROR"</span><span class="p">,</span><span class="w">
    </span><span class="nl">"message"</span><span class="p">:</span><span class="w"> </span><span class="s2">"The request contains invalid data"</span><span class="p">,</span><span class="w">
    </span><span class="nl">"details"</span><span class="p">:</span><span class="w"> </span><span class="p">[</span><span class="w">
      </span><span class="p">{</span><span class="w">
        </span><span class="nl">"field"</span><span class="p">:</span><span class="w"> </span><span class="s2">"price"</span><span class="p">,</span><span class="w">
        </span><span class="nl">"message"</span><span class="p">:</span><span class="w"> </span><span class="s2">"Price must be a positive number"</span><span class="w">
      </span><span class="p">},</span><span class="w">
      </span><span class="p">{</span><span class="w">
        </span><span class="nl">"field"</span><span class="p">:</span><span class="w"> </span><span class="s2">"isbn"</span><span class="p">,</span><span class="w">
        </span><span class="nl">"message"</span><span class="p">:</span><span class="w"> </span><span class="s2">"ISBN format is invalid"</span><span class="w">
      </span><span class="p">}</span><span class="w">
    </span><span class="p">]</span><span class="w">
  </span><span class="p">}</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre></div></div>

<h3 id="nested-resources-can-get-messy">Nested Resources Can Get Messy</h3>

<p>Be careful with deep nesting. This starts to look ridiculous:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>GET /api/v1/publishers/123/authors/456/books/789/reviews/101/comments/202
</code></pre></div></div>

<p>Instead, consider flattening:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>GET /api/v1/reviews/101/comments
GET /api/v1/comments?reviewId=101
</code></pre></div></div>

<h2 id="keep-in-mind">Keep in mind…</h2>

<p>Look, AI tools are great for generating code quickly, but they often miss these important facts. They’ll happily generate endpoints like <code class="language-plaintext highlighter-rouge">/api/getBookById</code> because they’re trained on examples that include bad APIs too.</p>

<p>Next time you’re working with an AI assistant, give it these guidelines upfront. Tell it to use proper REST conventions, meaningful HTTP methods, and consistent naming. Your future self (and your teammates) will thank you.</p>

<p>And remember, a well designed API is like a good conversation. It should be predictable, clear, and not leave people guessing what you meant.</p>]]></content><author><name>lpsandaruwan</name></author><category term="Posts" /><category term="REST" /><category term="API designing" /><category term="Software Engineering" /><category term="AI Assisted coding" /><category term="posts" /><category term="api" /><category term="restful" /><category term="software" /><category term="ai" /><summary type="html"><![CDATA[Here’s something that bugs me: AI tools like ChatGPT, Claude, Gemini, or whatever AI you’re using are great at cranking out code fast, but they often generate REST APIs that look like they were designed by someone who never had to maintain them.]]></summary></entry><entry><title type="html">Arch Linux for Developers: A Step-by-Step Guide to Building Your Ultimate Dev Environment</title><link href="https://lpsandaruwan.github.io/posts/archlinux-2024-guide/" rel="alternate" type="text/html" title="Arch Linux for Developers: A Step-by-Step Guide to Building Your Ultimate Dev Environment" /><published>2024-10-27T00:00:00+00:00</published><updated>2024-10-27T00:00:00+00:00</updated><id>https://lpsandaruwan.github.io/posts/archlinux-2024-guide</id><content type="html" xml:base="https://lpsandaruwan.github.io/posts/archlinux-2024-guide/"><![CDATA[<p><img src="/assets/images/content/posts/arch/arch.png" alt="arch logo" /></p>

<h2 id="why-arch-linux-lets-jump-in">Why Arch Linux? Let’s Jump In!</h2>

<p>Hey there, fellow developer community! If you’ve been browsing Linux distributions, you know there are <strong>tons</strong> of options out there. I’ve tested a few, gone on some Linux adventures, and finally landed on Arch Linux. And, wow, it’s been my go-to for over 11 years now. If you want a powerful development setup and want to learn Linux hands-on, Arch could be your golden ticket. Let’s explore why!</p>

<h2 id="starting-out-with-linux">Starting Out with Linux</h2>

<p>Before you dive into Arch Linux, get your feet wet with a few beginner-friendly options! Test out Linux Mint, Ubuntu, or Fedora on a virtual machine (like QEMU, VirtualBox, or VMware Player). These will help you get a feel for Linux without jumping straight into the deep end. Here’s what you’ll need to be comfortable with before going full Arch:</p>

<ul>
  <li><strong>Terminal Confidence</strong>: Yep, you’ll be typing commands often, so get cozy with the command line.</li>
  <li><strong>OS Basics</strong>: Understand simple things like what a kernel is or why drivers matter.</li>
  <li><strong>Hardware Awareness</strong>: Know a bit about your processor, GPU, network card, and RAM.</li>
  <li><strong>Experimenting Spirit</strong>: Don’t worry—trying things out is part of the fun, and mistakes won’t blow up your machine!</li>
</ul>

<h2 id="key-things-for-choosing-your-linux-distro">Key Things for Choosing Your Linux Distro</h2>

<p>When you’re picking a Linux distro, think about a few important things. The image below breaks down what makes up a Linux operating system.</p>

<p><img src="/assets/images/content/posts/arch/linux.drawio.png" alt="Linux internals" /></p>

<h3 id="1-hardware-compatibility">1. Hardware Compatibility</h3>

<p>Your hardware and the Linux kernel need to play nice together:</p>

<ul>
  <li><strong>New Hardware?</strong> Go for the latest LTS (Long-Term Support) or mainline kernel to avoid compatibility headaches.</li>
  <li><strong>Buying Gear?</strong> Avoid hardware with tricky proprietary drivers. Look up compatibility (especially wireless cards by Broadcom, Mediatek) before buying.</li>
  <li><strong>Struggling with Hardware?</strong> If Wi-Fi’s not working, try connecting via Ethernet or using USB tethering, Wi-Fi dongles, or modems to stay online.</li>
</ul>

<h3 id="2-types-of-linux-releases">2. Types of Linux Releases</h3>

<p>Distros handle updates in different ways, so here’s a quick rundown:</p>

<ul>
  <li><strong>Point Releases</strong>: Updates on a set schedule, like Fedora or Debian Stable.</li>
  <li><strong>LTS</strong>: Long-term support versions (like Ubuntu LTS), with years of support.</li>
  <li><strong>Rolling Releases</strong>: Continuous updates with no version numbers, like Arch and OpenSUSE Tumbleweed.</li>
  <li><strong>Semi-Rolling</strong>: Updates in chunks, like Manjaro.</li>
  <li><strong>Testing Releases</strong>: Think of these as beta versions, like Fedora Rawhide.</li>
  <li><strong>Enterprise/Corporate</strong>: Rock-solid, long-term releases, like Red Hat.</li>
</ul>

<h3 id="3-package-managers">3. Package Managers</h3>

<p>Every distro has a package manager—tools like <code class="language-plaintext highlighter-rouge">apt</code>, <code class="language-plaintext highlighter-rouge">yum</code>, <code class="language-plaintext highlighter-rouge">dnf</code>, and <code class="language-plaintext highlighter-rouge">pacman</code> help install and update software. The beauty of Linux is, if the package you want isn’t available, you can usually find a binary or build it yourself!</p>

<h3 id="4-desktop-environments-des">4. Desktop Environments (DEs)</h3>

<p>Linux has a ton of different desktop environments (DEs). Popular ones include KDE, GNOME, XFCE, and Cinnamon. Try out different DEs and find one that fits your style.</p>

<h2 id="why-arch-linux-heres-why-i-love-it">Why Arch Linux? Here’s Why I Love It</h2>

<p>After all my trials, I stuck with Arch Linux. Here’s why it’s been amazing for me:</p>

<h3 id="1-fresh-software-always">1. Fresh Software Always</h3>

<p>Arch Linux has the latest software, always. That means you get all the new features, bug fixes, and patches as soon as they’re available. I always use decent gaming laptops for power and portability(compared to Apple logos their price to specs ratio is top notch), so having the latest kernel updates keeps everything going smoothly.</p>

<h3 id="2-community-power">2. Community Power</h3>

<p>Arch is powered by its community. The Arch forums are full of wisdom, and the Arch Wiki is a treasure of guides and troubleshooting tips. The Arch Wiki is so good that even users of other distros rely on it!</p>

<h3 id="3-rolling-release">3. Rolling Release</h3>

<p>Arch is a rolling release, so you’re always on the latest version. No re-installs—just constant updates that keep your system fresh.</p>

<h3 id="4-pacman-package-manager">4. Pacman Package Manager</h3>

<p>Pacman is Arch’s package manager. It’s fast, simple, and handles dependencies smoothly. Just type a few commands, and Pacman does the rest.</p>

<h3 id="5-aur-arch-user-repository">5. AUR: Arch User Repository</h3>

<p>The AUR (Arch User Repository) is a huge collection of community-maintained scripts for software. If you can’t find something in the official repos, it’s probably in the AUR. Tools like <code class="language-plaintext highlighter-rouge">yay</code>, <code class="language-plaintext highlighter-rouge">pacaur</code>, or <code class="language-plaintext highlighter-rouge">octopi</code> make it easy to install AUR packages.</p>

<h3 id="6-zero-bloat">6. Zero Bloat</h3>

<p>Arch only has what you choose to install—no unnecessary pre-installed software, just the stuff you actually want and need.</p>

<h3 id="7-build-your-own-os-vibes">7. Build-Your-Own OS Vibes</h3>

<p>With Arch, you’re the creator. Setting it up lets you build a custom OS tailored to your needs. It’s challenging but super rewarding, and there’s nothing like the feeling of running an OS you crafted yourself!</p>

<hr />

<h2 id="lets-begin-">Let’s Begin 🚀</h2>
<p>Alright, you’ve got the scoop on why Arch Linux is worth it, so let’s start cooking our very own operating system!</p>

<h3 id="step-1-boot-up-the-arch-linux-installer-️">Step 1: Boot Up the Arch Linux Installer 🖥️</h3>

<ol>
  <li><strong>Download the ISO</strong>: Head over to the <a href="https://archlinux.org/download/">official Arch Linux download page</a> and grab the latest installation <code class="language-plaintext highlighter-rouge">.iso</code> file. This file will help us get started!</li>
  <li><strong>Create Bootable Media</strong>:
    <ul>
      <li><strong>On Windows</strong>: Use tools like Rufus, YUMI, or Ventoy to make your USB drive bootable.</li>
      <li><strong>On Linux or WSL</strong>: You can use the <code class="language-plaintext highlighter-rouge">dd</code> command to write the ISO to a USB drive.</li>
      <li><strong>On a Virtual Machine</strong>: Just load the ISO directly into your virtual machine.</li>
    </ul>
  </li>
</ol>

<blockquote>
  <p><strong>Note:</strong> If your hard drive is already full, you might need to shrink an existing partition to create some unallocated space for installing Arch. About 40GB should be more than enough.</p>
</blockquote>

<ol>
  <li>
    <p><strong>Boot It Up</strong>: Insert your bootable media into your PC and restart it. You’ll see the installer menu—choose the first option to kick things off!</p>

    <p><img src="/assets/images/content/posts/arch/installer.startup.png" alt="Startup" /></p>
  </li>
</ol>

<p>Now, you should see a root terminal waiting for you.</p>

<p><img src="/assets/images/content/posts/arch/startup.root.terminal.png" alt="Startup initial" /></p>

<hr />

<h3 id="step-2-configure-network-">Step 2: Configure Network 🌐</h3>
<p>We need an internet connection to download the packages that will build our system. Let’s find the network interface!</p>

<h4 id="list-network-interfaces">List Network Interfaces</h4>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">ls</span> /sys/class/net
</code></pre></div></div>
<blockquote>
  <p>This command lists all network interfaces. You might see something like:</p>
  <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>enp12s0  lo  wlp0s20f3
</code></pre></div>  </div>
  <ul>
    <li><code class="language-plaintext highlighter-rouge">lo</code> - Ignore this one; it’s just a virtual loopback device for local communication (like talking to yourself; localhost, 127.0.0.1 …).</li>
  </ul>
</blockquote>

<h4 id="if-youre-wired-ethernet">If You’re Wired (Ethernet)</h4>
<p>If you have a cable plugged in, you’re good to go! No extra steps needed!</p>

<h4 id="if-youre-on-wifi">If You’re on WiFi</h4>
<p>The <code class="language-plaintext highlighter-rouge">wlp*</code> device is likely your WiFi. Let’s confirm it:</p>
<blockquote>
  <p>(<code class="language-plaintext highlighter-rouge">dmesg</code> can show logs for device initializations at the kernel level.)</p>
  <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>dmesg | <span class="nb">grep</span> <span class="nt">-i</span> wlan
</code></pre></div>  </div>
</blockquote>

<p>We’ll use <code class="language-plaintext highlighter-rouge">iwd</code> to set up WiFi. Run these commands in sequence:</p>

<ol>
  <li>Start the <code class="language-plaintext highlighter-rouge">iwd</code> Interactive Shell:
    <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>iwctl
</code></pre></div>    </div>
  </li>
  <li>List Devices:
    <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>device list
</code></pre></div>    </div>
  </li>
  <li>Power On WiFi (if it’s off):
    <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>device wlan0 set-property Powered on <span class="c"># wlan0 is the name of the WiFi interface</span>
</code></pre></div>    </div>
    <blockquote>
      <p>If there’s an error, check if it’s blocked by <code class="language-plaintext highlighter-rouge">rfkill</code>:
(<code class="language-plaintext highlighter-rouge">rfkill</code> is a tool for managing radio devices like Bluetooth and WiFi; it helps with security and blocking issues.)</p>
      <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rfkill
</code></pre></div>      </div>
    </blockquote>
  </li>
  <li>If it’s blocked, unblock it and try turning the device on again:
    <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rfkill unblock wlan <span class="o">&amp;&amp;</span> device wlan0 set-property Powered on
</code></pre></div>    </div>
  </li>
  <li>Scan for Networks:
    <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>station wlan0 get-networks
</code></pre></div>    </div>
  </li>
  <li>Connect to Your Network (enter your WiFi password when prompted):
    <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>station wlan0 connect <span class="s2">"Network_SSID"</span>
</code></pre></div>    </div>
  </li>
  <li>Check Connectivity:
Let’s do a quick test to see if we’re connected:
    <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>ping <span class="nt">-c</span> 3 google.com
</code></pre></div>    </div>
    <p>And that’s it! Your network is set up, and we’re ready to install Arch. 🎉</p>
  </li>
</ol>

<hr />

<h3 id="step-3-preparing-disk-partitions-️">Step 3: Preparing Disk Partitions 🗂️</h3>
<p>Alright, let’s get ready to create some disk partitions! This is super important because a tiny mistake can wipe out your data. We’ll use a tool called <code class="language-plaintext highlighter-rouge">cfdisk</code> that gives us a nice visual interface to manage our disks.</p>

<h4 id="understanding-linux-file-system">Understanding Linux File System</h4>
<p>Before we dive in, let’s take a quick peek at what the Linux file system looks like. We’re going to create something similar on our hard drive!</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/
├── bin         # Important command files (like ls, cp, mv)
├── boot        # Boot files for starting up your system
├── dev         # Device files (like hard drives and USBs)
├── etc         # Configuration files for your system
├── home        # User home directories (like /home/user)
├── lib         # Essential libraries for commands
├── media       # Mount points for USB drives and CDs
├── mnt         # Temporary mount points
├── opt         # Optional software and packages
├── proc        # System information
├── root        # Home directory for the root user
├── run         # Temporary files since the last boot
├── sbin        # System administration commands
├── srv         # Service-related data
├── sys         # Kernel and system information
├── tmp         # Temporary files cleared on reboot
├── usr         # User-installed programs and libraries
│   ├── bin     # User-installed command files
│   ├── lib     # Shared libraries for user commands
│   └── share   # Shared files and docs (like icons)
└── var         # Variable data (like logs and databases)
    ├── log     # Log files
    ├── cache   # Cached data
    ├── lib     # Variable library files
    └── tmp     # App-created temporary files
</code></pre></div></div>

<h4 id="1-identify-your-disk-">1. Identify Your Disk 🧐</h4>
<p>First, we need to find out which disk we want to install Linux on. The command below will show you the disks and their sizes:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>lsblk
</code></pre></div></div>

<p>You’ll see something like this:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
nvme1n1     259:0    0 953.9G  0 disk
nvme0n1     259:10   0 476.9G  0 disk
</code></pre></div></div>

<h4 id="2-open-cfdisk-for-partitioning-">2. Open <code class="language-plaintext highlighter-rouge">cfdisk</code> for Partitioning 🔧</h4>
<p>Now, let’s open <code class="language-plaintext highlighter-rouge">cfdisk</code> to create our partitions. If your hard drive is <code class="language-plaintext highlighter-rouge">nvme1n1</code>, use this command:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cfdisk /dev/nvme1n1
</code></pre></div></div>

<p>This will pop up a window where you can create partitions. Here’s what you need to make:</p>

<ul>
  <li><strong>Root Partition</strong> (<code class="language-plaintext highlighter-rouge">/</code>) for the main system</li>
  <li><strong>Swap Partition</strong> for extra memory</li>
  <li><strong>EFI Boot Partition</strong> if your computer uses UEFI (most modern PCs do)</li>
  <li>You can also create a separate partition for your home directory (<code class="language-plaintext highlighter-rouge">/home</code>) if you like.</li>
</ul>

<p>For example, if you have about 163GB free, you could create:</p>
<ul>
  <li>60GB for root (<code class="language-plaintext highlighter-rouge">/</code>)</li>
  <li>90GB for home (<code class="language-plaintext highlighter-rouge">/home</code>)</li>
  <li>3GB for swap (helps when your RAM is full)</li>
  <li>800MB for EFI (if using UEFI)</li>
</ul>

<p>Once you’ve created the partitions, select the <code class="language-plaintext highlighter-rouge">Write</code> option in <code class="language-plaintext highlighter-rouge">cfdisk</code> to save your changes and exit.</p>

<h4 id="3-format-your-partitions-">3. Format Your Partitions 🎨</h4>
<p>Now, it’s time to format those partitions! We’ll use <code class="language-plaintext highlighter-rouge">EXT4</code> for the root and home partitions, and <code class="language-plaintext highlighter-rouge">FAT32</code> for the EFI partition. Let’s format:</p>

<p>First, check your partitions again with <code class="language-plaintext highlighter-rouge">lsblk</code>:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>lsblk
</code></pre></div></div>

<p>You’ll see your newly created partitions listed. Now, format them:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mkfs.ext4 /dev/nvme1n1p6 <span class="c"># Format root partition</span>
mkfs.ext4 /dev/nvme1n1p7 <span class="c"># Format home directory</span>
mkfs.vfat <span class="nt">-F32</span> /dev/nvme1n1p8 <span class="c"># Format EFI partition</span>
mkswap /dev/nvme1n1p9 <span class="c"># Set up Linux swap</span>
</code></pre></div></div>

<h4 id="4-mount-your-partitions-️">4. Mount Your Partitions 🏔️</h4>
<p>Time to mount those formatted disks! This will prepare them for the Arch Linux installation. Run these commands:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>mount /dev/nvme1n1p6 /mnt <span class="c"># Mount root partition</span>
<span class="nb">mkdir</span> <span class="nt">-p</span> /mnt/home <span class="o">&amp;&amp;</span> mount /dev/nvme1n1p7 /mnt/home <span class="c"># Mount home directory</span>
<span class="nb">mkdir</span> <span class="nt">-p</span> /mnt/boot/efi <span class="o">&amp;&amp;</span> mount /dev/nvme1n1p8 /mnt/boot/efi <span class="c"># Mount EFI partition</span>
swapon /dev/nvme1n1p9 <span class="c"># Enable swap</span>
</code></pre></div></div>

<p>Check your mount points one more time to ensure everything is set up correctly:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>lsblk
</code></pre></div></div>

<p>Your output should look something like this:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
nvme1n1     259:0    0 953.9G  0 disk
├─nvme1n1p6 259:6    0    90G  0 part /home
├─nvme1n1p7 259:7    0    60G  0 part /
├─nvme1n1p8 259:8    0   800M  0 part /boot/efi
└─nvme1n1p9 259:9    0     3G  0 part <span class="o">[</span>SWAP]
nvme0n1     259:10   0 476.9G  0 disk
</code></pre></div></div>

<p>Congratulations! Your partitions are all set and ready for the Arch Linux installation. 🎊 Let’s move on to the next step!</p>

<hr />

<h3 id="step-4-installing-core-linux-packages-and-configurations-">Step 4: Installing Core Linux Packages and Configurations 🐧</h3>

<p>Now, it’s time to bring your Arch Linux to life! We’ll install the essential core packages that make up the Linux system using the <code class="language-plaintext highlighter-rouge">pacstrap</code> command. This command will set up the Linux kernel and other important tools on your mounted partitions. Let’s get started!</p>

<p><strong>Install the core packages:</strong></p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pacstrap <span class="nt">-K</span> /mnt base linux linux-firmware <span class="se">\</span>
linux-headers base-devel nano <span class="nb">sudo</span> <span class="se">\</span>
intel-ucode <span class="se">\ </span><span class="c"># use only if you have an Intel CPU</span>
amd-ucode <span class="se">\ </span><span class="c"># only for AMD CPU</span>
</code></pre></div></div>

<p><em>(If you’re not sure about your CPU type, you can check it by running this command:)</em></p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>lscpu | <span class="nb">grep</span> <span class="nt">-i</span> <span class="s2">"Model name"</span>
</code></pre></div></div>

<p><strong>Generate <code class="language-plaintext highlighter-rouge">fstab</code>:</strong>
This step creates a file called <code class="language-plaintext highlighter-rouge">fstab</code> that tells Linux how to mount disk partitions when it starts up. Let’s make it!</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>genfstab <span class="nt">-U</span> /mnt <span class="o">&gt;&gt;</span> /mnt/etc/fstab
</code></pre></div></div>

<p><strong>Check your <code class="language-plaintext highlighter-rouge">fstab</code>:</strong>
To make sure everything is in order, let’s verify that all your partitions are listed in <code class="language-plaintext highlighter-rouge">fstab</code>. You can do this by running:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cat</span> /mnt/etc/fstab
</code></pre></div></div>

<p><strong>Change into the new system:</strong>
Now we need to switch into the mounted file system so we can start making configurations. We do this with the <code class="language-plaintext highlighter-rouge">arch-chroot</code> command.</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>arch-chroot /mnt
</code></pre></div></div>

<p><strong>Set up localization settings:</strong>
Next, we’ll set the language and locale for your system. Open the file <code class="language-plaintext highlighter-rouge">/etc/locale.gen</code> and find the line <code class="language-plaintext highlighter-rouge">LANG=en_US.UTF-8</code>. Uncomment it (just remove the <code class="language-plaintext highlighter-rouge">#</code> at the start) and feel free to uncomment any other localizations you want.</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nano /etc/locale.gen
</code></pre></div></div>

<p>After editing, run this command to generate the locales:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>locale-gen
</code></pre></div></div>

<p><strong>Set the computer name:</strong>
Let’s give your system a name! This name will identify your computer on the network. You can replace <code class="language-plaintext highlighter-rouge">myhostname</code> with whatever name you like.</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">echo </span>myhostname <span class="o">&gt;&gt;</span> /etc/hostname
</code></pre></div></div>

<p><strong>Set a password for the root user:</strong>
Now, let’s secure your system by setting a password for the root user.</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>passwd
</code></pre></div></div>

<p><strong>Create a new user:</strong>
It’s a good idea to create a normal user account for everyday tasks. This user will have some extra privileges. Just replace <code class="language-plaintext highlighter-rouge">"your username"</code> with your chosen name.</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>useradd <span class="nt">-m</span> <span class="nt">-g</span> <span class="nb">users</span> <span class="nt">-G</span> wheel,storage,power,audio <span class="nt">-s</span> /bin/bash <span class="s2">"your username"</span>
</code></pre></div></div>

<p><strong>Set a password for your new user:</strong></p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>passwd <span class="s2">"your username"</span>
</code></pre></div></div>

<p><strong>Give the new user sudo privileges:</strong>
Now, we want this user to be able to run commands as a superuser (the admin). To do that, we’ll edit the <code class="language-plaintext highlighter-rouge">sudo</code> configuration:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nv">EDITOR</span><span class="o">=</span>nano visudo
</code></pre></div></div>

<p>In the file, find the line that says <code class="language-plaintext highlighter-rouge">%wheel ALL=(ALL:ALL) ALL</code> and uncomment it (again, just remove the <code class="language-plaintext highlighter-rouge">#</code>). Then, save the changes.</p>

<p><strong>Install and enable Network Manager:</strong>
Finally, let’s install the network manager tools. This will help your system discover and connect to networks when it starts up.</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pacman <span class="nt">-S</span> networkmanager
systemctl <span class="nb">enable </span>NetworkManager
</code></pre></div></div>

<p>Now you’re all set! You’ve installed the core packages and configured your system. Your Arch Linux is well on its way to becoming fully functional! 🎉</p>

<hr />

<h3 id="step-5-install-bootloader-">Step 5: Install Bootloader 🚀</h3>

<p>There are many bootloaders out there, but I chose <strong>GRUB</strong>. Why? Because it’s mature and has lots of features!</p>

<ol>
  <li><strong>First, install the required packages.</strong>
    <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pacman <span class="nt">-S</span> grub <span class="se">\</span>
os-prober <span class="se">\</span>
efibootmgr <span class="c"># only if the system supports UEFI</span>
</code></pre></div>    </div>
  </li>
  <li><strong>Now, install GRUB.</strong>
If you have a <strong>UEFI</strong> system, run:
    <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>grub-install <span class="nt">--target</span><span class="o">=</span>x86_64-efi <span class="nt">--efi-directory</span><span class="o">=</span>/boot/efi <span class="nt">--bootloader-id</span><span class="o">=</span>GRUB
</code></pre></div>    </div>

    <p>If you have an older <strong>legacy</strong> system, use this command:</p>
    <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>grub-install <span class="nt">--target</span><span class="o">=</span>i386-pc /dev/xxx <span class="c"># xxx is the storage name.</span>
</code></pre></div>    </div>
  </li>
  <li><strong>Generate configurations.</strong>
If you are dual-booting with <strong>Windows</strong>, find the line that says <code class="language-plaintext highlighter-rouge">GRUB_DISABLE_OS_PROBER=false</code> in the default GRUB configuration file and uncomment it.
    <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nano /etc/default/grub
</code></pre></div>    </div>

    <p>Next, generate the GRUB configuration.</p>
    <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>grub-mkconfig <span class="nt">-o</span> /boot/grub/grub.cfg
</code></pre></div>    </div>
  </li>
</ol>

<hr />

<h3 id="step-6-installing-a-desktop-environment-️">Step 6: Installing a Desktop Environment 🖼️</h3>

<p>A desktop environment (DE) gives you a friendly user interface to interact with your Linux system. There are many DEs to choose from. <strong>KDE</strong> is a complete and feature-rich option. If you prefer something simpler, you might like <strong>GNOME</strong>, <strong>Cinnamon</strong>, or <strong>XFCE</strong>. This setup focuses on <strong>KDE</strong>.</p>

<p>To install KDE, run:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pacman <span class="nt">-S</span> plasma-meta kde-system kde-utilities <span class="se">\</span>
kde-graphics <span class="c"># optional software</span>
</code></pre></div></div>

<p>After that, enable the login manager that comes with the plasma-meta package group:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>systemctl <span class="nb">enable </span>sddm
</code></pre></div></div>

<p>Now everything is ready! Exit the <code class="language-plaintext highlighter-rouge">chroot</code> session, unmount the disks, and restart your system:</p>
<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">exit
</span>umount <span class="nt">-R</span> /mnt
swapoff /dev/<span class="s2">"swap partition name"</span>
reboot
</code></pre></div></div>

<p>When your system boots up, you will be greeted by your new desktop environment!</p>

<p><img src="/assets/images/content/posts/arch/installation_finished.png" alt="after_login" /></p>

<hr />

<h3 id="step-7-post-install-packages-️">Step 7: Post Install Packages 🛠️</h3>

<p>Alright, let’s pimp up your Arch Linux setup with some essential packages! Here’s how you can make your system more powerful with extra tools and dev environments.</p>

<h4 id="1-add-aur-support">1. Add AUR Support</h4>

<p>The <a href="https://aur.archlinux.org/">AUR (Arch User Repository)</a> is a huge collection of community-maintained scripts for installing software that’s not available in the default <code class="language-plaintext highlighter-rouge">pacman</code> repositories. You can either download and run these scripts yourself or use a tool to make it super easy. My go-to choice is <a href="https://github.com/Jguer/yay"><code class="language-plaintext highlighter-rouge">yay</code></a>, because it works just like <code class="language-plaintext highlighter-rouge">pacman</code>—easy to remember! Other popular options are <code class="language-plaintext highlighter-rouge">pacaur</code>, <code class="language-plaintext highlighter-rouge">aurman</code>, <code class="language-plaintext highlighter-rouge">aura</code>, <code class="language-plaintext highlighter-rouge">pacseek</code>, etc.</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c"># Install yay</span>
<span class="nb">sudo </span>pacman <span class="nt">-S</span> <span class="nt">--needed</span> git base-devel <span class="o">&amp;&amp;</span> git clone https://aur.archlinux.org/yay.git <span class="o">&amp;&amp;</span> <span class="nb">cd </span>yay <span class="o">&amp;&amp;</span> makepkg <span class="nt">-si</span>
</code></pre></div></div>

<p>With <code class="language-plaintext highlighter-rouge">yay</code>, you can:</p>
<ul>
  <li><strong>Update the system</strong>: <code class="language-plaintext highlighter-rouge">yay</code></li>
  <li><strong>Search for a package</strong>: <code class="language-plaintext highlighter-rouge">yay -Ss xyz</code></li>
  <li><strong>Install a package</strong>: <code class="language-plaintext highlighter-rouge">yay -S xyz</code></li>
  <li><strong>Uninstall a package</strong>: <code class="language-plaintext highlighter-rouge">yay -Rns xyz</code></li>
  <li><strong>Force remove</strong> (use with caution): <code class="language-plaintext highlighter-rouge">yay -Rdd xyz</code></li>
  <li><strong>Clean unused dependencies</strong>: <code class="language-plaintext highlighter-rouge">yay -Yc</code></li>
  <li><strong>Show remote package info</strong>: <code class="language-plaintext highlighter-rouge">yay -Si xyz</code></li>
  <li><strong>Show local package info</strong>: <code class="language-plaintext highlighter-rouge">yay -Qi xyz</code></li>
  <li><strong>List all installed packages</strong>: <code class="language-plaintext highlighter-rouge">yay -Qq</code></li>
  <li><strong>List explicitly installed packages</strong>: <code class="language-plaintext highlighter-rouge">yay -Qqe</code></li>
</ul>

<h4 id="2-proprietary-drivers">2. Proprietary Drivers</h4>

<p>If you have a laptop with hybrid graphics (like Intel + Nvidia or AMD + Nvidia), you’ll want to get the right drivers. To check what graphics cards you have, run:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>lspci | <span class="nb">grep</span> <span class="nt">-E</span> <span class="s1">'VGA|3D'</span>
</code></pre></div></div>

<p>Then, install the drivers you need:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>pacman <span class="nt">-S</span> mesa xf86-video-intel         <span class="c"># For Intel GPUs</span>
<span class="nb">sudo </span>pacman <span class="nt">-S</span> nvidia nvidia-utils nvidia-settings  <span class="c"># For Nvidia GPUs</span>
<span class="nb">sudo </span>pacman <span class="nt">-S</span> mesa xf86-video-amdgpu        <span class="c"># Open-source AMD driver</span>
</code></pre></div></div>

<p>If you’ve got Nvidia hybrid graphics, you can install <code class="language-plaintext highlighter-rouge">envycontrol</code> to easily switch between GPUs.</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>yay <span class="nt">-S</span> envycontrol
</code></pre></div></div>

<p>To list or set the GPU mode:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>envycontrol <span class="nt">--query</span>            <span class="c"># Show current GPU</span>
envycontrol <span class="nt">--switch</span> nvidia     <span class="c"># Options: integrated, hybrid, nvidia</span>
</code></pre></div></div>

<h4 id="3-install-a-web-browser">3. Install a Web Browser</h4>

<p>Choose your favorite browser(s) and install them:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>yay <span class="nt">-S</span> google-chrome           <span class="c"># Google Chrome</span>
pacman <span class="nt">-S</span> chromium             <span class="c"># Open-source version of Chrome</span>
pacman <span class="nt">-S</span> firefox              <span class="c"># Mozilla Firefox</span>
yay <span class="nt">-S</span> opera                   <span class="c"># Opera browser</span>
</code></pre></div></div>

<hr />

<h3 id="step-8-setting-up-development-environments-">Step 8: Setting Up Development Environments 🧑‍💻</h3>

<p>Let’s get your coding environment all set up! Arch Linux is great for developers, and here’s how you can get started with various programming tools.</p>

<h4 id="1-java-development-kit-jdk">1. Java Development Kit (JDK)</h4>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>yay <span class="nt">-Ss</span> jdk                     <span class="c"># List available JDK versions</span>
yay <span class="nt">-S</span> jdk-lts                  <span class="c"># Install long-term support (LTS) version, or specify another version if you need</span>
</code></pre></div></div>

<p>To check or set up Java versions:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>archlinux-java status           <span class="c"># Show installed JDKs and the current default</span>
archlinux-java <span class="nb">set</span> &lt;JAVA_ENV_NAME&gt; <span class="c"># Set default Java version</span>
</code></pre></div></div>

<h4 id="2-python-development">2. Python Development</h4>

<p>First, install <code class="language-plaintext highlighter-rouge">pyenv</code> to manage different Python versions.</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>pacman <span class="nt">-S</span> pyenv
pyenv <span class="nb">install </span>3.8               <span class="c"># Install Python 3.8.x</span>
pyenv <span class="nb">install </span>3.11              <span class="c"># Install Python 3.11.x</span>
pyenv global 3.11               <span class="c"># Set Python 3.11 as the default</span>
</code></pre></div></div>

<h4 id="3-nodejs-environment">3. Node.js Environment</h4>

<p>Install <code class="language-plaintext highlighter-rouge">nvm</code> (Node Version Manager) first:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>yay <span class="nt">-S</span> nvm
</code></pre></div></div>

<p>Add <code class="language-plaintext highlighter-rouge">nvm</code> paths to <code class="language-plaintext highlighter-rouge">~/.bashrc</code>:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">echo</span> <span class="s1">'export NVM_DIR="$HOME/.nvm"'</span> <span class="o">&gt;&gt;</span> ~/.bashrc
<span class="nb">echo</span> <span class="s1">'[ -s "/usr/share/nvm/init-nvm.sh" ] &amp;&amp; \. "/usr/share/nvm/init-nvm.sh"'</span> <span class="o">&gt;&gt;</span> ~/.bashrc
<span class="nb">source</span> ~/.bashrc
</code></pre></div></div>

<p>Now you can install any Node version:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nvm <span class="nb">install </span>x.y.z               <span class="c"># Install Node version x.y.z</span>
nvm use x.y.z                   <span class="c"># Set current Node version</span>
node <span class="nt">--version</span>                  <span class="c"># Check the current Node version</span>
</code></pre></div></div>

<h4 id="4-ruby-development-environment">4. Ruby Development Environment</h4>

<p>Install <code class="language-plaintext highlighter-rouge">rbenv</code> for Ruby version management:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>yay <span class="nt">-S</span> rbenv
yay <span class="nt">-S</span> ruby-build               <span class="c"># Required for installing Ruby versions</span>
<span class="nb">echo</span> <span class="s1">'eval "$(rbenv init -)"'</span> <span class="o">&gt;&gt;</span> ~/.bashrc
<span class="nb">source</span> ~/.bashrc
</code></pre></div></div>

<p>To install and set up Ruby:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>rbenv <span class="nb">install </span>3.0.0
rbenv global 3.0.0              <span class="c"># Set Ruby 3.0.0 as default</span>
gem <span class="nt">-v</span>                          <span class="c"># Check that Ruby is correctly set up</span>
</code></pre></div></div>

<h4 id="5-net-environment">5. .NET Environment</h4>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>yay <span class="nt">-S</span> dotnet-install
dotnet-install <span class="nt">--channel</span> 8.0    <span class="c"># Install the .NET version 8.0</span>
</code></pre></div></div>

<h4 id="6-docker">6. Docker</h4>

<p>Install and set up Docker:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>pacman <span class="nt">-S</span> docker
<span class="nb">sudo </span>usermod <span class="nt">-aG</span> docker <span class="nv">$USER</span>   <span class="c"># Add your user to the Docker group</span>
<span class="nb">sudo </span>systemctl <span class="nb">enable </span>docker
</code></pre></div></div>

<p>After a system reboot, check that Docker is running:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>systemctl status docker
</code></pre></div></div>

<h4 id="7-database-tools">7. Database Tools</h4>

<p>For database management, here are some good gui tools:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>pacman <span class="nt">-S</span> dbeaver          <span class="c"># DBeaver, supports multiple databases</span>
yay <span class="nt">-S</span> pgadmin4-desktop         <span class="c"># Postgres admin tool</span>
yay <span class="nt">-S</span> sql-workbench            <span class="c"># MySQL Workbench</span>
yay <span class="nt">-S</span> mongodb-compass          <span class="c"># MongoDB management tool</span>
</code></pre></div></div>

<hr />

<h3 id="step-9-install-ides--️">Step 9: Install IDEs 💻 🛠️</h3>

<p>Now that you’ve set up the development kits you need, let’s get some awesome IDEs to write code!</p>

<h4 id="1-visual-studio-code">1. Visual Studio Code</h4>

<p>Quick and powerful, VS Code is a top pick for developers.</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>pacman <span class="nt">-S</span> code
</code></pre></div></div>

<h4 id="2-jetbrains-toolbox">2. JetBrains Toolbox</h4>

<p>Want options? JetBrains Toolbox lets you choose from their lineup of IDEs (including Android Studio!).</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>yay <span class="nt">-S</span> jetbrains-toolbox
</code></pre></div></div>

<h4 id="3-good-old-eclipse">3. Good Old Eclipse</h4>

<p>If you’re an Eclipse fan, check out these specialized versions for different languages.</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>yay <span class="nt">-S</span> spring-tool-suite      <span class="c"># For Spring framework (Java)</span>
yay <span class="nt">-S</span> eclipse-jee-bin        <span class="c"># Eclipse for Java EE</span>
yay <span class="nt">-S</span> eclipse-cpp-bin        <span class="c"># Eclipse for C/C++</span>
yay <span class="nt">-S</span> aptana-studio          <span class="c"># Eclipse-based IDE for PHP</span>
</code></pre></div></div>

<hr />

<h3 id="step-10-cloud-cli-tools-️">Step 10: Cloud CLI Tools ☁️</h3>

<p>Working with cloud services? These CLI tools will make managing them a breeze!</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>yay <span class="nt">-S</span> aws-cli-v2-bin         <span class="c"># AWS CLI, command: aws</span>
yay <span class="nt">-S</span> google-cloud-cli       <span class="c"># GCP CLI, command: gcloud</span>
<span class="nb">sudo </span>pacman <span class="nt">-S</span> azure-cli      <span class="c"># Microsoft Azure CLI, command: az</span>
</code></pre></div></div>
<hr />

<h3 id="step-11-miscellaneous-">Step 11: Miscellaneous 🎩</h3>

<h4 id="1-make-your-terminal-look-awesome-with-starship">1. Make Your Terminal Look Awesome with <code class="language-plaintext highlighter-rouge">starship</code></h4>

<p>Let’s make your terminal look stylish! <code class="language-plaintext highlighter-rouge">starship</code> is a cool tool to add colors, symbols, and more.</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pacman <span class="nt">-S</span> starship                  <span class="c"># Install starship</span>
<span class="nb">echo</span> <span class="s1">'eval "$(starship init bash)"'</span> <span class="o">&gt;&gt;</span> ~/.bashrc  <span class="c"># Load starship in each terminal</span>
<span class="nb">mkdir</span> <span class="nt">-p</span> ~/.config <span class="o">&amp;&amp;</span> <span class="nb">touch</span> ~/.config/starship.toml  <span class="c"># Create config file</span>
</code></pre></div></div>

<p>Now add this funky setup in the config file with:</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>nano ~/.config/starship.toml
</code></pre></div></div>

<p>Paste this sample configurations:</p>

<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>palette = "dracula"

[aws]
style = "bold orange"

[character]
error_symbol = "[λ](bold red)"
success_symbol = "[λ](bold green)"

[cmd_duration]
style = "bold yellow"

[directory]
style = "bold green"

[git_branch]
style = "bold pink"

[git_status]
style = "bold red"

[hostname]
disabled = false
ssh_only = false
trim_at = "."
style = "bold green"

[username]
format = "[$user]($style) on "
style_user = "bold yellow"
show_always = true
style_root = "bold red"

[sudo]
symbol = '🧙 '
style = "bold red"
format = '[as $symbol]($style)'
disabled = false

[palettes.dracula]
background = "#282a36"
current_line = "#44475a"
foreground = "#f8f8f2"
comment = "#6272a4"
cyan = "#8be9fd"
green = "#50fa7b"
orange = "#ffb86c"
pink = "#ff79c6"
purple = "#bd93f9"
red = "#ff5555"
yellow = "#f1fa8c"
</code></pre></div></div>

<p>Close and open your terminal to see the magical transformation!</p>

<h4 id="2-entertainment-music-and-videos">2. Entertainment: Music and Videos</h4>

<p>Time for some fun! Here are great media players for your audio and video needs:
(There is plenty: vlc, clementine, deadbeef, mplayer, strawberry, amarok, …)</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">sudo </span>pacman <span class="nt">-S</span> audacious smplayer   <span class="c"># Audacious for music, SMPlayer for videos</span>
</code></pre></div></div>

<p>Want system-wide sound effects? Try JamesDSP for an enhanced audio experience.</p>

<div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>yay <span class="nt">-S</span> jamesdsp
</code></pre></div></div>

<p><img src="/assets/images/content/posts/arch/jamesdsp.png" alt="JamesDSP Screenshot" /></p>

<h4 id="3-communication-apps">3. Communication Apps</h4>

<p>Stay connected with friends, family, or your team using these popular apps:</p>

<ul>
  <li><strong>Microsoft Teams for Linux</strong>:
    <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>  yay <span class="nt">-S</span> teams
</code></pre></div>    </div>
  </li>
  <li><strong>Slack</strong>:
    <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>  yay <span class="nt">-S</span> slack-desktop
</code></pre></div>    </div>
  </li>
  <li><strong>Discord</strong>:
    <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>  <span class="nb">sudo </span>pacman <span class="nt">-S</span> discord
</code></pre></div>    </div>
  </li>
  <li><strong>Skype</strong>:
    <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code>  yay <span class="nt">-S</span> skypeforlinux-bin
</code></pre></div>    </div>
  </li>
</ul>

<h4 id="4-disable-file-indexing-kde-only">4. Disable File Indexing (KDE Only)</h4>

<p>KDE’s file indexing can sometimes slow things down. Here’s how to disable it:</p>

<ol>
  <li>Open <strong>System Settings</strong>.</li>
  <li>Go to <strong>Workspace Options &gt; File Search</strong>.</li>
  <li>Unselect <strong>File Indexing</strong> and click <strong>Apply</strong>.</li>
</ol>

<h4 id="5-fix-missing-windows-entry-in-dual-boot-grub-menu">5. Fix Missing Windows Entry in Dual Boot Grub Menu</h4>

<p>If Windows doesn’t show up in the Grub menu, follow these steps:</p>

<ol>
  <li>Open your Windows partitions in the file manager (this mounts them).</li>
  <li>
    <p>Then update Grub:</p>

    <div class="language-shell highlighter-rouge"><div class="highlight"><pre class="highlight"><code> grub-mkconfig <span class="nt">-o</span> /boot/grub/grub.cfg
</code></pre></div>    </div>
  </li>
</ol>

<hr />

<p>With these tools and customizations, you’ve transformed your Arch Linux into a powerhouse ready for anything—coding, gaming, creating, or just enjoying a sleek, personalized experience. You’re now in control of a system built exactly the way you want, with the power to expand, modify, and customize as you grow. Arch Linux isn’t just an OS; it’s a launchpad for learning and exploring your own potential. So go ahead, break things, fix them, and build something amazing. This is your machine, your rules—now let’s see what incredible things you can do with it!</p>]]></content><author><name>lpsandaruwan</name></author><category term="Posts" /><category term="Linux" /><category term="OS" /><category term="Development" /><category term="posts" /><category term="linux" /><category term="os" /><category term="development" /><summary type="html"><![CDATA[]]></summary></entry><entry><title type="html">GrooveCraft: Building Your Own Amplifier for Next-Level Music Vibes</title><link href="https://lpsandaruwan.github.io/posts/crafting-audio-ecosystem/" rel="alternate" type="text/html" title="GrooveCraft: Building Your Own Amplifier for Next-Level Music Vibes" /><published>2023-12-23T00:00:00+00:00</published><updated>2023-12-23T00:00:00+00:00</updated><id>https://lpsandaruwan.github.io/posts/crafting-audio-ecosystem</id><content type="html" xml:base="https://lpsandaruwan.github.io/posts/crafting-audio-ecosystem/"><![CDATA[<blockquote class="prompt-warning">
  <p>People have different preferences when it comes to how they like to listen to music, and the sound experience can be different for each person. Here, I’ve put together my findings on creating an affordable do-it-yourself sound system that suits my taste.</p>
</blockquote>

<p><img src="/assets/images/content/posts/diy-amp/wave-nero-forte.png" alt="wave_graph_nero_forte_slipknot" /></p>

<h2 id="what-makes-sound-good">What Makes Sound Good?</h2>

<p>Okay, so everyone’s got their own taste in sound. Some love the deep, thumping bass that makes your heart go boom. Others are into the high, sparkly notes that tingle your ears. Then there are folks who like adding cool effects, like digital magic or echoes. And, of course, some just want the original sound without any changes. It’s like picking your favorite flavor of music – everyone’s got their own sweet spot! 🎵🔊</p>

<h2 id="exploring-ways-to-enhance-sound-quality">Exploring Ways to Enhance Sound Quality.</h2>

<h3 id="1-quality-of-audio-data">1. Quality of Audio Data.</h3>

<p>The sound you hear is like a digital superhero! It’s stored as digital data, kind of like a musical code. The best quality comes from something called lossless music data. It means the sound stays exactly as it was recorded – no changes. There are also cool codes like AAC and MP3, with more bits, higher sample rates, and deeper depths they can also shine. Honestly, my ears can’t catch the difference in AAC music beyond a sample rate of 48kHz, bit rate of ~256kbps, and depth of 24 bits. If you can, your ears might be super sensitive!</p>

<h3 id="2-media-players">2. Media Players.</h3>

<p>Choosing a music player is like picking the coolest DJ for your tunes. You’ve got a bunch of them for different operating systems. Each player uses its own algorithm for handling digital music data. Since I mostly roll with Linux and Android, my picks are Apple Music for Android (Loseless music streaming), for Linux Aqualung, Audacious, and Deadbeef. And on Windows, Windows Media Player is my top pick. 🎶✨</p>

<p><img src="/assets/images/content/posts/diy-amp/loseless_stream.jpg" alt="loseless_streaming" /></p>

<h3 id="3-supercharging-your-music-digital-signal-processing-plugins-dsp-plugins">3. Supercharging Your Music: Digital Signal Processing Plugins (DSP Plugins).</h3>

<p>Want to spice up your music experience? You can play around with some cool tools called Digital Signal Processing (DSP) plugins. The superstar among them is the Equalizer, letting you boost the frequencies you love. But there’s more to explore – plugins like surround, reverb, echo, harmonics, compressors, and even frequency filtering and cutoffs.</p>

<p>On Linux my go-to buddies are JamesDSP4Linux (a nifty pipewire DSP tool) and Calf Studio Gear (especially when using Jack2, mostly for line-in audio). I personally enjoy adding a bit of treble boost, lowering the mid-range a bit(1kHz as the low-lying), and keeping the bass reduced or flat for those low-frequency vibes. It is a modified “V” curved equalization effect with frequencies decreasing from 500Hz to around 4kHz. It’s a very pleasing tone for my ears! 🎶✨</p>

<ul>
  <li>
    <p>JamesDSP4Linux
<img src="/assets/images/content/posts/diy-amp/eq_v_curve.png" alt="jamesdsp" /></p>
  </li>
  <li>
    <p>Calf studio gear</p>
  </li>
</ul>

<p><img src="/assets/images/content/posts/diy-amp/eq_calf.png" alt="calf" /></p>

<h3 id="4-the-dac-turning-digital-to-analog">4. The DAC: Turning Digital to Analog.</h3>

<p>It’s a crucial gadget for turning the digital tunes into real analog sound. I went for a USB DAC from Creative Labs (Model SB1560, Omni Surround 5.1) in my setup. Despite being an older model, the output is impressively superior and clearer compared to other DACs I’ve experimented with, including those bundled with Realtek, CMedia like chips and other integrated/budget-friendly sound cards. 🎧✨</p>

<h3 id="5-the-pre-amplifier-pumping-up-the-dac-signal">5. The Pre Amplifier: Pumping Up the DAC signal.</h3>

<p>So, after the DAC finishes its digital-to-analog conversion, the preamplifier steps in. It grabs those signals from the DAC and boosts the electric signal from the DAC if needed, especially when receiving a weak signal. Now, the way it enhances audio can vary, thanks to the design of the preamp. Most preamp circuits come with a gain controller, bass level, treble level, and optionally, left and right channel balance or mid-range frequency adjustment controllers. Personally, I prefer sticking with three-band control parameters or none at all since I found it makes no sense to change the balance or signal level between left and right channels.</p>

<h3 id="6-power-amplifier---electrify-the-audio-signal">6. Power Amplifier - Electrify The Audio Signal.</h3>
<p>The power amp takes the enhanced signal from the preamplifier and boosts it up big time. It’s the magic wand that cranks it up, increases the gain to make the signal stronger, and the sound louder.</p>

<h3 id="7-speakers-the-sound-performer">7. Speakers: The Sound Performer.</h3>

<p>Speakers need to be loud enough, cover a wide range of frequencies to capture every sound detail, and smoothly reproduce those frequencies. My preference is separate speakers for different frequency ranges. This is beneficial because bass frequencies can sometimes overpower the mids and highs at higher volumes when using only a full-range speaker, causing issues like Doppler distortion and draining power for bass.</p>

<p>Modern full-range speakers attempt to address these issues, but they can be a bit pricey. However, investing in quality performers is like having front-row seats to your favorite concert! 🎶✨</p>

<h3 id="8-speaker-positioning-listening-zone-configuration">8. Speaker Positioning: Listening Zone Configuration.</h3>

<p>How you place your speakers and set up your room can totally switch up how your ear catch those frequencies. The position of your speakers and the size of your room may change how you feel the music vibes.</p>

<p>For instance, if you tuck your bass buddy (woofer) in a room corner, get ready for some serious bass punch. But, be sneaky with those high-frequency speakers; if you cover them up, you might miss out on those crystal-clear notes you love.</p>

<p>Now, here’s my sweet spot: If I’ve got two cool bookshelf speakers, I’d pop them in front, one on the left and one on the right, just below ear level – it’s like the music is chatting right at me. And if there’s a subwoofer in the mix, I’d sneak it into the back corner of the room.</p>

<p>It’s like finding the perfect, comfortable spot to watch a movie!</p>

<h3 id="9-ultimately-take-care-of-your-ears">9. Ultimately: Take Care of Your Ears</h3>

<p>A rocking music system won’t do much if you forget about your ear health. Here are some simple tips that I follow to keep the listening ability tip-top:</p>

<ul>
  <li>Dial Down the Volume: Keep the music vibe alive by lowering the volume for those marathon music sessions.</li>
  <li>Give Your Ears a Timeout: Treat your ears to some quiet time. Find a peaceful spot and let your ears recover from all that sound stimulation.</li>
  <li>Mix It Up: Surprise your ears! Explore different music genres and styles to keep things interesting.</li>
  <li>Stay Hydrated and Rest Well: Good health equals happy ears. Keep hydrated, catch some quality Zs – your body’s auditory system will thank you.</li>
  <li>Follow the 60/60 Rule: If you’re rocking headphones or earbuds, here’s a cool rule: take a break every 60 minutes, especially if the volume is cranked up past 60% of the maximum. Your ears deserve a breather!</li>
</ul>

<h3 id="10-other-important-aspects-of-sound-processing">10. Other Important Aspects of Sound Processing.</h3>

<h4 id="thd---total-harmonic-distortion">THD - Total Harmonic Distortion.</h4>
<ul>
  <li>Imagine a game: lower THD means your sound triumphs with less distortion.</li>
  <li>It’s the gauge of how much your sound might embark on a distortion adventure under different conditions.</li>
</ul>

<h4 id="snr---signal-to-noise-ratio">SNR - Signal to Noise Ratio.</h4>
<ul>
  <li>Envision a superhero showdown: higher SNR means your signal is a superhero, warding off noise villains.</li>
  <li>It’s the measure of how much unwanted noise wants to sneak into your audio signal.</li>
</ul>

<h4 id="electric-signal-distribution">Electric Signal Distribution.</h4>
<ul>
  <li>The wires – the unsung heroes of the audio world!</li>
  <li>Using high-resistance wires is like sending your frequencies through a tricky maze. I go for wires with less than ~0.2 ohms resistance – smooth transfering of electric signals.</li>
</ul>

<h4 id="amplifier-class">Amplifier Class</h4>

<ul>
  <li><strong>Class A:</strong> The noble knight of low distortion, though not the most power-efficient.</li>
  <li><strong>Class B:</strong> Efficient powerhouse, but watch out for the crossover distortion ninja.</li>
  <li><strong>Class AB:</strong> Strikes a balance, reducing crossover distortion compared to its ninja cousin, Class B.</li>
  <li><strong>Class C:</strong> The highly efficient speedster, perfect for less distortion-critical missions.</li>
  <li><strong>Class D:</strong> The efficiency maestro, often found rocking out in portable audio devices.</li>
  <li><strong>Class H:</strong> The zen master, balancing efficiency and distortion – the wise choice for professional audio amplifiers.</li>
</ul>

<h2 id="the-groovy-diy-fix-enhancing-the-analog-configuration">The Groovy DIY Fix! Enhancing The Analog Configuration.</h2>

<p>I’ve already delved into selecting audio data, DSPs, and DACs above. Now, let’s dive into my adventure of deciphering the analog configuration for the audio.</p>

<p><img src="/assets/images/content/posts/diy-amp/high_level_graph.jpg" alt="high_level_graph" /></p>

<p><img src="/assets/images/content/posts/diy-amp/low_level_graph.jpg" alt="high_level_graph" /></p>

<h3 id="1-the-speakers">1. The Speakers.</h3>

<p>I scored this awesome classic Sony 5.1 speaker set online for a steal! These older speakers are not just easy on the wallet but also deliver a super clear and lively sound, way better than those budget Chinese speakers flooding the market(no offence). But I prefer a simple stereo (2 channels or 2.1) sound vibe. No need for fancy 4.0, 5.1, or 7.1 setups. So, I decided to move ahead with just the front speakers (SS-TS31) and the subwoofer box (SS-WS31) from this set.</p>

<p><img src="/assets/images/content/posts/diy-amp/sony-spks.jpg" alt="sony_speakers" /></p>

<h3 id="2-the-pre-amplifier">2. The Pre-Amplifier.</h3>

<p>As a heavy metal enthusiast, finding the right preamp for my intense tunes was a challenge. The music is loud, aggressive, and the vocals are brutal. Here’s the scoop on the preamp setups I put to the test because most online reviews are based on user preferences.</p>

<p>I used Audio Technica ATH-M20X headphones (not the best, but they deliver solid clear flat frequency) and lossless music playback to find the right one. Here’s the lowdown:</p>

<p><strong>The Soundtrack of Judgment:</strong></p>

<ul>
  <li><strong>Nero Forte by Slipknot:</strong> Pure metal chaos from start to finish. Heavy vocals, pounding beats.</li>
  <li><strong>Blood, Tears, Dust by Lacuna Coil:</strong> Aggressive dual vocalists, aggressive melody – metal with a dramatic twist.</li>
  <li><strong>Lost in the Echo by Linkin Park:</strong> An all-time favorite alternative rock song with electronic vibes.</li>
  <li><strong>Blinding Lights by The Weeknd:</strong> Pop with a touch of heart in the accompaniments.</li>
  <li><strong>Nothing Else Matters by Metallica:</strong> Need I say more?</li>
  <li><strong>Interstellar Theme by Hans Zimmer:</strong> No words, just goosebumps.</li>
</ul>

<p><strong>The Pre-Amplifier Compilation:</strong></p>

<p>While listening to the above songs, I took each circuit for a spin and rated them based on my metal standards. Some circuits I soldered myself, and some I grabbed already soldered(below pre-amplifier circuits are very much available, affordable and cheap). Your preferences might vary, so give them a shot!</p>

<table>
  <thead>
    <tr>
      <th>IC/Transistor</th>
      <th>Score</th>
      <th>Sound at Max Gain</th>
      <th>Heavy Metal</th>
      <th>Low-Noise Genres</th>
      <th>Vocals Clarity</th>
      <th>Accompaniment Clarity</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>LM324</td>
      <td>12</td>
      <td>2</td>
      <td>2</td>
      <td>3</td>
      <td>2</td>
      <td>3</td>
    </tr>
    <tr>
      <td>LM358</td>
      <td>12</td>
      <td>2</td>
      <td>2</td>
      <td>3</td>
      <td>2</td>
      <td>3</td>
    </tr>
    <tr>
      <td>NE5532</td>
      <td>16</td>
      <td>3</td>
      <td>3</td>
      <td>4</td>
      <td>3</td>
      <td>3</td>
    </tr>
    <tr>
      <td>TDA 1524A</td>
      <td>14</td>
      <td>2</td>
      <td>3</td>
      <td>4</td>
      <td>3</td>
      <td>3</td>
    </tr>
    <tr>
      <td>UA741</td>
      <td>12</td>
      <td>2</td>
      <td>2</td>
      <td>3</td>
      <td>2</td>
      <td>3</td>
    </tr>
    <tr>
      <td>C945</td>
      <td>17</td>
      <td>4</td>
      <td>3</td>
      <td>4</td>
      <td>3</td>
      <td>3</td>
    </tr>
  </tbody>
</table>

<ul>
  <li><strong>IC/Transistor:</strong> The brain of the preamp. Results may vary due to the overall circuit design, so experiment away!</li>
  <li><strong>Sound at Maximum Gain:</strong> Cranked the preamp to the max. Let’s see what they got.</li>
  <li><strong>Heavy Metal Music:</strong> Complicated, noisy music playback.</li>
  <li><strong>Low Noisy Music:</strong> Everything else – pop, electronic, RnB, you name it.</li>
  <li><strong>Vocals Clarity:</strong> Can I hear those guttural screams clearly?</li>
  <li><strong>Accompaniment Clarity:</strong> The hidden gems – harmonics, rhythm guitars, electronic beats, etc.</li>
</ul>

<p><strong>Scores:</strong></p>
<ul>
  <li><strong>1:</strong> Useless.</li>
  <li><strong>2:</strong> Okay, some distortion at high volumes, but bearable.</li>
  <li><strong>3:</strong> Good, barely noticeable distortion when the gain is up.</li>
  <li><strong>4:</strong> Excellent, no crackles, no distortion, and no overlapping sounds.</li>
</ul>

<p>Based on my research, I picked the pre-amplifier circuit with <strong>NE5532</strong>.</p>
<ul>
  <li>It performs well on gain without adjusting or normalizing the frequency levels.</li>
  <li>Instead of balancing stereo channels (which makes no sense to me), it has a mid-range frequency controlling knob.</li>
  <li>No hiss sound until it is more than ~90% of the maximum gain level.</li>
  <li>No buzzing sounds even when the transformer is close by.</li>
  <li>It has good reviews on the internet.</li>
</ul>

<h3 id="3-the-power-amplifier">3. The Power Amplifier.</h3>

<p>Just like with pre-amplifiers, I explored amplifier chips and circuit setups to find the perfect match to power up the signal for my speakers. With a listening space of about 144 sqft, I aimed for an amplifier delivering 20 to 100 watts, ideal for my Sony speakers with a 3-ohm impedance and a ~130 watts rating (not exceeding 4 ohms and 50 watts per channel, keeping it safe). Since I roll with a 2.1 speaker setup, I needed a 3-channel amplification output.</p>

<p><img src="/assets/images/content/posts/diy-amp/tda_amp.jpg" alt="tda_amp" /></p>

<h4 id="amplifier-configurations">Amplifier Configurations</h4>

<p>To configure the 3-channel output, I utilized multiple pre-amplifiers with band controls. Here’s the breakdown:</p>

<ul>
  <li><strong>NE5532 for Left/Right Channels:</strong>
    <ul>
      <li>Set the bass knob to 0, eliminating the need for a high pass filter.</li>
    </ul>
  </li>
  <li><strong>TDA 1524A for Subwoofer Channel:</strong>
    <ul>
      <li>Turned the treble knob to 0, complemented by a passive low pass filter with a resistor and capacitor, restricting subwoofer output up to ~300Hz.</li>
    </ul>
  </li>
</ul>

<p><img src="/assets/images/content/posts/diy-amp/low_pass.png" alt="low_pass" /></p>

<p><strong>1. Ripped-off Circuit from Microlab M100 Subwoofer System (Class - Probably D):</strong></p>
<ul>
  <li>Integrated speaker quality was terrible, lacking bass and high frequencies.</li>
  <li>Performed decently with Sony speakers.</li>
  <li>Used NE5532 pre-amplifier only as this circuit is already 2.1.</li>
  <li>Moved on due to insufficient power.</li>
</ul>

<p><strong>2. Stereo LA4440 + Mono LA4440 (Class - AB, Felt More Like Class B):</strong></p>
<ul>
  <li>A classic amplifier design.</li>
  <li>Sound resembled an old radio.</li>
  <li>Quickly moved on.</li>
</ul>

<p><strong>3. Mono Channel TDA2005 x 3 (Class - AB, The Hot Plate):</strong></p>
<ul>
  <li>TDA 2005 is obsolete but available.</li>
  <li>Unpleasant sound with no clarity.</li>
  <li>ICs heated up excessively, even burning my finger accidentally.</li>
  <li>Chips eventually exploded with smoke.</li>
  <li>Probably low-quality replicas.</li>
</ul>

<p><strong>4. Stereo TDA2030A + Mono TDA2030A (Class - AB, Better Than Nothing!):</strong></p>
<ul>
  <li>Also obsolete but available.</li>
  <li>Good sound profile, slight crackling at high gain levels.</li>
  <li>Did not heat like TDA2005.</li>
  <li>Likely high-quality replicas per online reviews.</li>
  <li>Unsatisfied with high-frequency clarity, moved on.</li>
</ul>

<p><strong>5. Stereo LM1875 + Mono LM1875 (Class AB, Finally a Good Player!):</strong></p>
<ul>
  <li>Tested Texas Instrument’s substitute for TDA 5-pin IC.</li>
  <li>Replaced TDA2030s with LM1875 in existing circuits.</li>
  <li>Very good sound quality, better than TDA2030A.</li>
  <li>Slight sound crackling at higher gain levels.</li>
  <li>ICs heated up more than TDA2030A.</li>
  <li>Subwoofer channel frequently shut down in longer listening sessions, likely due to integrated thermal protection.</li>
  <li>Decided to move on without adding a larger radiator.</li>
</ul>

<p><strong>6. Dual TPA3116D2, 2.1 Circuit (Class D, An All-Inclusive Hero to the Rescue):</strong></p>
<ul>
  <li>Tried another plug-and-play circuit, also from Texas Instruments with excellent reviews.</li>
  <li>Efficient, higher sound level even at ear-piercing high frequencies and heart-attack-level bass, yet the radiator barely got warm or stayed cool.</li>
  <li>No complaints about sound quality; remarkable clarity for a small circuit.</li>
  <li>Lower THD, integrated speaker guard, and wider operating voltage range were added advantages.</li>
  <li>Sealed the deal with this amplifier.</li>
</ul>

<p><img src="/assets/images/content/posts/diy-amp/circuit.jpg" alt="final" /></p>

<blockquote class="prompt-tip">
  <p>Please disregard the unsightly black tapes; they’re there due to my phobia and past experiences with circuit-roasting short circuits. 😂</p>
</blockquote>

<blockquote class="prompt-warning">
  <p>I removed the passive L/R high pass circuits from the above design as they masked the high frequencies I desired, regardless of the values I used for the capacitor and resistor according to the formula <strong>fc = 1/2πRC</strong>.</p>
</blockquote>

<h2 id="final-conclusion">Final Conclusion</h2>

<p>Considering all the research and money spent, I could have bought a decent off-the-shelf speaker system. But where’s the fun in that? 😎</p>

<p><img src="/assets/images/content/posts/diy-amp/amp_dac.jpg" alt="closing" /></p>]]></content><author><name>lpsandaruwan</name></author><category term="Posts" /><category term="Audio" /><category term="Hobby" /><category term="Music" /><category term="audio" /><category term="music" /><category term="hobby" /><category term="electronics" /><summary type="html"><![CDATA[People have different preferences when it comes to how they like to listen to music, and the sound experience can be different for each person. Here, I’ve put together my findings on creating an affordable do-it-yourself sound system that suits my taste.]]></summary></entry><entry><title type="html">Saga Pattern with serverless model on Google Cloud Platform - Part 1</title><link href="https://lpsandaruwan.github.io/posts/saga-gcp-choreography/" rel="alternate" type="text/html" title="Saga Pattern with serverless model on Google Cloud Platform - Part 1" /><published>2022-11-20T00:00:00+00:00</published><updated>2022-11-20T00:00:00+00:00</updated><id>https://lpsandaruwan.github.io/posts/saga-gcp-choreography</id><content type="html" xml:base="https://lpsandaruwan.github.io/posts/saga-gcp-choreography/"><![CDATA[<blockquote class="prompt-info">
  <p>Gostep: 👉 <a href="https://github.com/gostep-cli/gostep/wiki">Guide</a><br />
Materials: 👉 <a href="https://github.com/lpsandaruwan/saga-gcp">Complete source code</a></p>
</blockquote>

<p>During the past few years, the microservices architecture(MSA) and serverless model have gained a lot of popularity in the industry. However, these technologies come with their own set of challenges. One substantial challenge is managing data in MSA due to its complexity. Considering common patterns for MSA data management we will be focusing on the Saga pattern in this article.</p>

<h2 id="the-saga-pattern">The Saga pattern</h2>
<p>In order to manage business transactions across multiple microservices, the Saga pattern was introduced. Basically it is a series of local transactions; every transaction happens within the boundary of the micro-service, which every service will publish an event after the transaction for the next subsequent micro-service to perform the next transaction consuming the published event. This process will continue till the last transaction. In case any transaction failed in this series Saga will execute a series of fallback actions to undo the impact of all previous transactions.</p>

<p>There are two approaches to implementing the Saga pattern.</p>
<ul>
  <li>
    <p><strong>Choreography</strong> - The micro-service is responsible for emitting events eventually of its local transaction. The published event will trigger the execution of local transactions in microservices subscribed to the event. Also in this approach micro-service is responsible for handling the errors.</p>
  </li>
  <li>
    <p><strong>Orchestration</strong> - A central orchestrator(a stateful coordinator) will trigger the local transactions in services and will maintain the global transaction status including handling errors.</p>
  </li>
</ul>

<p>Now that we have a basic understanding of Saga pattern, we will discuss how to implement Saga pattern, defining an example for both approaches using Google Cloud Serverless model.</p>

<h2 id="the-real-world-example">The real world example</h2>
<p>Let’s consider a train ticket booking system.</p>

<p><img src="/assets/images/content/posts/gcp-saga/flowchart.png" alt="flowchart" /></p>

<p>The workflow consists of,</p>

<ol>
  <li>Send a seat reservation request</li>
  <li>Check for available transits in the database and proceed with seat booking.</li>
  <li>Hold the number of seats until payment is processed.</li>
  <li>Process the payment.</li>
  <li>Confirm the seat booking.</li>
  <li>Confirm the reservation and notify the customer.</li>
</ol>

<p>However if the system encountered any error while running a local transaction, the fallback sequence should be executed to undo all the changes happening in the global transaction to keep the <a href="https://en.wikipedia.org/wiki/ACID">ACID</a> properties.</p>

<h2 id="preparing-the-development-environment">Preparing the development environment</h2>

<blockquote class="prompt-warning">
  <p>(Please note that we won’t be using a real payment gateway or a notification service, beacause the main purpose of this article is to demonstrate how to use severless model for Saga.)</p>
</blockquote>

<p>To implement the solution we will be using Google Cloud serverless services, MongoDb and Javascript.</p>

<p>Before we begin we must have,</p>
<ul>
  <li>A billing enabled Google Cloud project</li>
  <li>Prior knowledge in Google Cloud Services</li>
  <li>Python, NodeJs, GCloud cli tools installed in your system(If you are using Windows, WSL might come in handy)</li>
</ul>

<h3 id="google-cloud-clicloud-console">Google Cloud CLI/Cloud console</h3>
<p>You can use both CLI tools or web console to create and modify services. In this article we will be mostly using CLI tools. Please follow https://cloud.google.com/sdk/docs/install to install the Google Cloud SDK. And once you installed the SDK run <code class="language-plaintext highlighter-rouge">gcloud init</code> command and follow instructions to configure credentials.</p>

<h3 id="building-the-cloud-functions-project-structure">Building the Cloud functions project structure</h3>

<p>To build the project structure and functions, we will be using <strong>gostep</strong>, a pythonic CLI tool that I created previously to manage implementations when there are a lot of cloud functions.</p>

<p>To use gostep you need to have <strong>Subversion CLI</strong>, <strong>Python version 3</strong> and <strong>Pip</strong> package manager installed(Setup a virtual environment of your own preference). When you are ready, run the command, <code class="language-plaintext highlighter-rouge">pip install gostep</code>. For more information please refer, http://lahirus.com/gostep-intro. Also please make sure that you have enabled Cloud build APIs(https://console.cloud.google.com/apis/library/cloudbuild.googleapis.com).</p>

<p>Using <code class="language-plaintext highlighter-rouge">gostep</code>, let’s first create a Cloud Functions project.</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">mkdir </span>SagaChoreography <span class="o">&amp;&amp;</span> <span class="nb">cd </span>SagaChoreography
gostep auth init reservationsservice       <span class="c"># Creates a service account credentials file</span>
gostep base init reservations location asia-east2 version <span class="s2">"0.1.0"</span>   <span class="c"># Creates gostep project metadata files and directory structure.</span>
</code></pre></div></div>

<p>Now we can have the project base. Let’s move ahead with implementing local transactions and services.</p>

<h2 id="choreography-based-solution">Choreography based solution</h2>

<p>For the demonstration we will be using,</p>
<ul>
  <li>Pub/Sub for event sharing</li>
  <li>Firestore to store event data</li>
  <li>MongoDb as the transits service database</li>
</ul>

<p><img src="/assets/images/content/posts/gcp-saga/sagachoreographyflow.png" alt="saga chor flow" /></p>

<h3 id="transits-service">Transits service</h3>

<p>This micro-service is responsible for CRUD operations on train entities.</p>

<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">// Transit document schema</span>
  <span class="p">{</span>
       <span class="nl">transitId</span><span class="p">:</span> <span class="nx">string</span><span class="p">,</span>
       <span class="nx">trainName</span><span class="p">:</span> <span class="nx">string</span><span class="p">,</span>
       <span class="nx">start</span><span class="p">:</span> <span class="nx">string</span><span class="p">,</span>
       <span class="nx">destination</span><span class="p">:</span> <span class="nx">string</span><span class="p">,</span>
       <span class="nx">day</span><span class="p">:</span> <span class="nx">string</span><span class="p">,</span>
       <span class="nx">departure</span><span class="p">:</span> <span class="nx">number</span><span class="p">,</span>
       <span class="nx">arrival</span><span class="p">:</span> <span class="nx">number</span><span class="p">,</span>
       <span class="nx">availableSeats</span><span class="p">:</span> <span class="nx">number</span><span class="p">,</span>
       <span class="nx">lockedSeats</span><span class="p">:</span> <span class="nx">number</span><span class="p">,</span>
       <span class="nx">totalSeats</span><span class="p">:</span> <span class="nx">number</span>
  <span class="p">}</span>
</code></pre></div></div>
<p>As the database, we will be using <a href="https://console.cloud.google.com/marketplace/product/mongodb/mdb-atlas-self-service">MongoDB Atlas pay as you go</a> service in the GCP marketplace After configuring the MongoDb instance, let’s create the transits function.</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gostep service init transits version <span class="s2">"0.1.0"</span> <span class="nb">env </span>nodejs
</code></pre></div></div>

<p>This will create a boilerplate NodeJs cloud function in {PROJECT_ROOT}/src/transits and it can be executed as a http request after the deployment.</p>

<p>Now let’s include the dependencies.</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cd </span>src/transits
npm <span class="nb">install</span> <span class="nt">--save</span> mongodb
</code></pre></div></div>

<p>After creating the transits database and the collection, we can add MongoDb connection URI and collection name in the <code class="language-plaintext highlighter-rouge">src/trains/functions.json</code> as an environment variable.</p>

<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="dl">"</span><span class="s2">environmentVariables</span><span class="dl">"</span><span class="p">:</span> <span class="p">{</span>
    <span class="dl">"</span><span class="s2">DB_URI</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">mongodb+srv://&lt;username&gt;:&lt;password&gt;@&lt;your-cluster-url&gt;/&lt;dbname&gt;</span><span class="dl">"</span><span class="p">,</span>
    <span class="dl">"</span><span class="s2">COLLECTION</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Transits</span><span class="dl">"</span>
    <span class="p">},</span>
</code></pre></div></div>

<p>First let’s use these environment variables and create a function to connect to the database.</p>

<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">import</span> <span class="p">{</span> <span class="nx">MongoClient</span> <span class="p">}</span> <span class="k">from</span> <span class="dl">"</span><span class="s2">mongodb</span><span class="dl">"</span><span class="p">;</span>
 

<span class="kd">const</span> <span class="nx">DB_URI</span> <span class="o">=</span> <span class="nx">process</span><span class="p">.</span><span class="nx">env</span><span class="p">.</span><span class="nx">DB_URI</span> <span class="o">||</span> <span class="dl">"</span><span class="s2">&lt;Default DB con URI&gt;</span><span class="dl">"</span><span class="p">;</span>
 
<span class="kd">const</span> <span class="nx">dbClient</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">MongoClient</span><span class="p">(</span><span class="nx">DB_URI</span><span class="p">);</span>
 
<span class="kd">const</span> <span class="nx">initDbClientConnection</span> <span class="o">=</span> <span class="k">async</span> <span class="p">()</span> <span class="o">=&gt;</span> <span class="p">{</span>
    <span class="k">try</span> <span class="p">{</span>
        <span class="k">await</span> <span class="nx">dbClient</span><span class="p">.</span><span class="nx">connect</span><span class="p">();</span>
    <span class="p">}</span> <span class="k">catch</span><span class="p">(</span><span class="nx">e</span><span class="p">)</span> <span class="p">{</span>
        <span class="nx">console</span><span class="p">.</span><span class="nx">error</span><span class="p">(</span><span class="nx">e</span><span class="p">);</span>
        <span class="k">throw</span> <span class="k">new</span> <span class="nb">Error</span><span class="p">(</span><span class="dl">"</span><span class="s2">Database failed to connect!</span><span class="dl">"</span><span class="p">);</span>
    <span class="p">}</span>
<span class="p">};</span>
</code></pre></div></div>

<p>And now let’s write 2 functions to find transits documents and save/update documents.</p>

<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">const</span> <span class="nx">COLLECTION</span> <span class="o">=</span> <span class="nx">process</span><span class="p">.</span><span class="nx">env</span><span class="p">.</span><span class="nx">COLLECTION</span> <span class="o">||</span> <span class="dl">"</span><span class="s2">Transits</span><span class="dl">"</span><span class="p">;</span>

<span class="kd">const</span> <span class="nx">query</span> <span class="o">=</span> <span class="k">async</span> <span class="p">(</span><span class="nx">queries</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="p">{</span>
    <span class="k">try</span> <span class="p">{</span>
        <span class="k">await</span> <span class="nx">initDbClientConnection</span><span class="p">();</span>
        <span class="kd">const</span> <span class="nx">transits</span> <span class="o">=</span> <span class="nx">dbClient</span><span class="p">.</span><span class="nx">db</span><span class="p">().</span><span class="nx">collection</span><span class="p">(</span><span class="nx">COLLECTION</span><span class="p">);</span>
        <span class="k">return</span> <span class="k">await</span> <span class="nx">transits</span><span class="p">.</span><span class="nx">find</span><span class="p">(</span><span class="nx">queries</span><span class="p">).</span><span class="nx">toArray</span><span class="p">();</span>
    <span class="p">}</span> <span class="k">catch</span> <span class="p">(</span><span class="nx">e</span><span class="p">)</span> <span class="p">{</span>
        <span class="nx">console</span><span class="p">.</span><span class="nx">error</span><span class="p">(</span><span class="nx">e</span><span class="p">);</span>
        <span class="k">throw</span> <span class="k">new</span> <span class="nb">Error</span><span class="p">(</span><span class="dl">"</span><span class="s2">Failed to query transits!</span><span class="dl">"</span><span class="p">)</span>
    <span class="p">}</span> <span class="k">finally</span> <span class="p">{</span>
        <span class="k">await</span> <span class="nx">dbClient</span><span class="p">.</span><span class="nx">close</span><span class="p">();</span>
    <span class="p">}</span>
<span class="p">}</span>

<span class="kd">const</span> <span class="nx">save</span> <span class="o">=</span> <span class="k">async</span><span class="p">(</span><span class="nx">transitId</span><span class="p">,</span> <span class="nx">patches</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="p">{</span>
    <span class="k">try</span> <span class="p">{</span>
        <span class="k">await</span> <span class="nx">initDbClientConnection</span><span class="p">();</span>
        <span class="kd">const</span> <span class="nx">transits</span> <span class="o">=</span> <span class="nx">dbClient</span><span class="p">.</span><span class="nx">db</span><span class="p">().</span><span class="nx">collection</span><span class="p">(</span><span class="nx">COLLECTION</span><span class="p">);</span>
        <span class="kd">const</span> <span class="nx">targetData</span> <span class="o">=</span> <span class="p">{</span> <span class="dl">"</span><span class="s2">$set</span><span class="dl">"</span><span class="p">:</span> <span class="nx">patches</span> <span class="p">};</span>
        <span class="k">await</span> <span class="nx">transits</span><span class="p">.</span><span class="nx">updateOne</span><span class="p">({</span> <span class="na">transitId</span><span class="p">:</span> <span class="nx">transitId</span> <span class="p">},</span> <span class="nx">targetData</span><span class="p">,</span> <span class="p">{</span> <span class="na">upsert</span><span class="p">:</span> <span class="kc">true</span> <span class="p">});</span>
    <span class="p">}</span> <span class="k">catch</span><span class="p">(</span><span class="nx">e</span><span class="p">)</span> <span class="p">{</span>
        <span class="nx">console</span><span class="p">.</span><span class="nx">error</span><span class="p">(</span><span class="nx">e</span><span class="p">);</span>
        <span class="k">throw</span> <span class="k">new</span> <span class="nb">Error</span><span class="p">(</span><span class="dl">"</span><span class="s2">Failed to update transits!</span><span class="dl">"</span><span class="p">);</span>
    <span class="p">}</span> <span class="k">finally</span> <span class="p">{</span>
        <span class="k">await</span> <span class="nx">dbClient</span><span class="p">.</span><span class="nx">close</span><span class="p">();</span>
    <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>

<p>In the main function we map GET and PUT https methods to above functions.</p>

<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">export</span> <span class="kd">const</span> <span class="nx">main</span> <span class="o">=</span> <span class="k">async</span> <span class="p">(</span><span class="nx">req</span><span class="p">,</span> <span class="nx">res</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="p">{</span>
    <span class="k">if</span><span class="p">(</span><span class="nx">req</span><span class="p">.</span><span class="nx">method</span> <span class="o">===</span> <span class="dl">"</span><span class="s2">GET</span><span class="dl">"</span><span class="p">)</span> <span class="p">{</span>
        <span class="nx">res</span><span class="p">.</span><span class="nx">json</span><span class="p">(</span><span class="k">await</span> <span class="nx">query</span><span class="p">(</span><span class="nx">req</span><span class="p">.</span><span class="nx">query</span><span class="p">));</span>
    <span class="p">}</span> <span class="k">else</span> <span class="k">if</span><span class="p">(</span><span class="nx">req</span><span class="p">.</span><span class="nx">method</span> <span class="o">===</span> <span class="dl">"</span><span class="s2">PUT</span><span class="dl">"</span><span class="p">)</span> <span class="p">{</span>
        <span class="kd">const</span> <span class="nx">transitId</span> <span class="o">=</span> <span class="nx">req</span><span class="p">.</span><span class="nx">query</span><span class="p">[</span><span class="dl">"</span><span class="s2">transitId</span><span class="dl">"</span><span class="p">];</span>
        <span class="k">if</span><span class="p">(</span><span class="o">!</span><span class="nx">transitId</span><span class="p">)</span> <span class="p">{</span>
            <span class="nx">res</span><span class="p">.</span><span class="nx">status</span><span class="p">(</span><span class="mi">400</span><span class="p">).</span><span class="nx">send</span><span class="p">({</span> <span class="dl">"</span><span class="s2">error</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Invalid parameters!</span><span class="dl">"</span> <span class="p">})</span>
        <span class="p">}</span>
        <span class="k">await</span> <span class="nx">save</span><span class="p">(</span><span class="nx">transitId</span><span class="p">,</span> <span class="nx">req</span><span class="p">.</span><span class="nx">body</span><span class="p">);</span>
        <span class="nx">res</span><span class="p">.</span><span class="nx">status</span><span class="p">(</span><span class="mi">201</span><span class="p">).</span><span class="nx">send</span><span class="p">();</span>
    <span class="p">}</span> <span class="k">else</span> <span class="p">{</span>
        <span class="nx">res</span><span class="p">.</span><span class="nx">status</span><span class="p">(</span><span class="mi">400</span><span class="p">).</span><span class="nx">json</span><span class="p">({</span> <span class="dl">"</span><span class="s2">error</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Invalid request</span><span class="dl">"</span> <span class="p">});</span>
    <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>

<p>Great! Now we have our transits service. We can deploy it by running below command in the project root,</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gostep deploy diff
</code></pre></div></div>

<p>After the deployment, transits service can be executed using http requests. But the endpoint is not available for the public. To test it locally, use the bearer token which you can obtain using the Gcloud cli.</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gcloud auth print-identity-token
</code></pre></div></div>

<h3 id="reservations-service">Reservations service</h3>

<p>Next, we are going to implement the entrypoint of the global transaction. Like before, let’s bootstrap a cloud function again. Run,</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gostep service init reservations version <span class="s2">"0.1.0"</span> <span class="nb">env </span>nodejs
</code></pre></div></div>

<p>Now we have our boilerplate code in <code class="language-plaintext highlighter-rouge">{PROJECT_ROOT}/src/reservations</code>.</p>

<p>Considering this scenario the <strong>reservations</strong> function is responsible for,</p>
<ol>
  <li>Get the user request via a HTTP request.</li>
  <li>Call transits service and find out if there is any transit avaialable.</li>
  <li>If a transit is avialable publish an message to the relavent topic.</li>
  <li>Save the event data with it’s status as ‘IN_PROGRES’, to update later.</li>
</ol>

<p>We are going to keep the event data stored in a database. So that we can keep the status of the particular event to use later. For that purpose we use Google Cloud firestore(data store in native mode), which is a serverless easy to use document database.
To enable Firestore run,</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gcloud firestore databases create <span class="nt">--region</span><span class="o">=</span>asia-southeast1
</code></pre></div></div>

<p>After that let’s install the dependencies. In the function root({PROJECT_ROOT}/src/resrevations) run,</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>npm <span class="nb">install</span> <span class="nt">--save</span> <span class="s2">"@google-cloud/firestore"</span> <span class="s2">"@google-cloud/pubsub"</span>
</code></pre></div></div>

<p>Let’s assume below payload as the request JSON.</p>
<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">{</span>
    <span class="dl">"</span><span class="s2">day</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Monday</span><span class="dl">"</span><span class="p">,</span>
    <span class="dl">"</span><span class="s2">start</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Colombo</span><span class="dl">"</span><span class="p">,</span>
    <span class="dl">"</span><span class="s2">destination</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">Ragama</span><span class="dl">"</span><span class="p">,</span>
    <span class="dl">"</span><span class="s2">numberOfSeats</span><span class="dl">"</span><span class="p">:</span> <span class="mi">10</span><span class="p">,</span>
    <span class="dl">"</span><span class="s2">userId</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">xyz@gmail.com</span><span class="dl">"</span>
<span class="p">}</span>
</code></pre></div></div>

<p>Once the user made his request we have to obtain the available transits for the requested day, start position and destination of the transit. To do that we will using a HTTP request to the <strong>transits</strong> service we implemeted before. Since the transits APIs are not publically available we have to use the <code class="language-plaintext highlighter-rouge">google-auth-library</code> to authorize requests from other services(<a href="https://cloud.google.com/functions/docs/securing/authenticating">See more</a>). There is no need to add the auth library as a dependecy since it is an already included library in the cloud function runtime.</p>

<p>First let’s add an environment variable for the transits API endpoint in <code class="language-plaintext highlighter-rouge">{PROJECT_ROOT}/src/transits/function.json</code>.</p>

<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code>    <span class="dl">"</span><span class="s2">environmentVariables</span><span class="dl">"</span><span class="p">:</span> <span class="p">{</span>
        <span class="dl">"</span><span class="s2">TRANSITS_API</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">{HOST_ADDRESS}/transits</span><span class="dl">"</span>
    <span class="p">}</span>
</code></pre></div></div>

<p>After that let’s authorize our request to fetch available transits.</p>

<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">import</span> <span class="p">{</span> <span class="nx">GoogleAuth</span> <span class="p">}</span> <span class="k">from</span> <span class="dl">"</span><span class="s2">google-auth-library</span><span class="dl">"</span><span class="p">;</span>


<span class="kd">const</span> <span class="nx">TRANSITS_API</span> <span class="o">=</span> <span class="nx">process</span><span class="p">.</span><span class="nx">env</span><span class="p">.</span><span class="nx">TRANSITS_API</span> <span class="o">||</span> <span class="dl">"</span><span class="s2">{DEFAULT_TRANSITS_HOST_ADDRESS}/transits</span><span class="dl">"</span><span class="p">;</span>

<span class="k">export</span> <span class="kd">const</span> <span class="nx">getAvailableTransits</span> <span class="o">=</span> <span class="k">async</span> <span class="p">(</span><span class="nx">numberOfSeats</span><span class="p">,</span> <span class="nx">day</span><span class="p">,</span> <span class="nx">destination</span><span class="p">,</span> <span class="nx">start</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="p">{</span>
    <span class="k">try</span><span class="p">{</span>
        <span class="c1">// Create an authorized client to invoke restricted Transits API.</span>
        <span class="kd">const</span> <span class="nx">auth</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">GoogleAuth</span><span class="p">();</span>
        <span class="kd">const</span> <span class="nx">transitsApiClient</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">auth</span><span class="p">.</span><span class="nx">getIdTokenClient</span><span class="p">(</span><span class="nx">TRANSITS_API</span><span class="p">);</span>

        <span class="kd">const</span> <span class="nx">result</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">transitsApiClient</span><span class="p">.</span><span class="nx">request</span><span class="p">({</span>
            <span class="na">url</span><span class="p">:</span> <span class="s2">`</span><span class="p">${</span><span class="nx">TRANSITS_API</span><span class="p">}</span><span class="s2">?day=</span><span class="p">${</span><span class="nx">day</span><span class="p">}</span><span class="s2">&amp;destination=</span><span class="p">${</span><span class="nx">destination</span><span class="p">}</span><span class="s2">&amp;start=</span><span class="p">${</span><span class="nx">start</span><span class="p">}</span><span class="s2">`</span><span class="p">,</span>
            <span class="na">method</span><span class="p">:</span> <span class="dl">"</span><span class="s2">GET</span><span class="dl">"</span>
        <span class="p">});</span>
        <span class="k">return</span> <span class="nx">result</span><span class="p">.</span><span class="nx">data</span><span class="p">.</span><span class="nx">filter</span><span class="p">(</span><span class="nx">element</span> <span class="o">=&gt;</span> <span class="nx">element</span><span class="p">[</span><span class="dl">"</span><span class="s2">availableSeats</span><span class="dl">"</span><span class="p">]</span> <span class="o">&gt;=</span> <span class="nx">numberOfSeats</span><span class="p">);</span>
    <span class="p">}</span> <span class="k">catch</span><span class="p">(</span><span class="nx">e</span><span class="p">)</span> <span class="p">{</span>
        <span class="nx">console</span><span class="p">.</span><span class="nx">error</span><span class="p">(</span><span class="nx">e</span><span class="p">);</span>
    <span class="p">}</span>
<span class="p">};</span>
</code></pre></div></div>

<p>Based on the result of the API request, we proceed further. Let’s assume that we got a list of available transits and we selected the topmost transit. Now we will be saving the event data in <strong>firestore</strong>, with a unique Id(a generated UUID as <code class="language-plaintext highlighter-rouge">correlationId</code>) as the global transation Id to identify the local transactions group and the status of the current event. It will aid to identify the local transaction for later references.</p>

<p>Same as before we can add the firestore collection name(reffered as <strong>kind</strong> in firestore) of the event as an environment varibale in <code class="language-plaintext highlighter-rouge">{PROJECT_ROOT}/src/transits/function.json</code>.</p>

<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code>    <span class="dl">"</span><span class="s2">environmentVariables</span><span class="dl">"</span><span class="p">:</span> <span class="p">{</span>
        <span class="dl">"</span><span class="s2">EVENT_DATA_COLLECTION</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">reservations</span><span class="dl">"</span>
    <span class="p">}</span>
</code></pre></div></div>

<p>Now we can write our function to save event data in firestore. Please note that you don’t have to include configurations to authorize the connection to the firestore since the cloud function runtime has the authorized access to the firestore in the same project.</p>

<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">import</span> <span class="nx">Firestore</span> <span class="k">from</span> <span class="dl">"</span><span class="s2">@google-cloud/firestore</span><span class="dl">"</span><span class="p">;</span>


<span class="kd">const</span> <span class="nx">EVENT_DATA_COLLECTION</span> <span class="o">=</span> <span class="nx">process</span><span class="p">.</span><span class="nx">env</span><span class="p">.</span><span class="nx">EVENT_DATA_COLLECTION</span> <span class="o">||</span> <span class="dl">"</span><span class="s2">reservations</span><span class="dl">"</span><span class="p">;</span>

<span class="k">export</span> <span class="kd">const</span> <span class="nx">saveEvent</span> <span class="o">=</span> <span class="k">async</span> <span class="p">(</span><span class="nx">id</span><span class="p">,</span> <span class="nx">eventData</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="p">{</span>
    <span class="k">try</span> <span class="p">{</span>
        <span class="kd">const</span> <span class="nx">firestore</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">Firestore</span><span class="p">();</span>
        <span class="kd">const</span> <span class="nx">docRef</span> <span class="o">=</span> <span class="nx">firestore</span><span class="p">.</span><span class="nx">collection</span><span class="p">(</span><span class="nx">EVENT_DATA_COLLECTION</span><span class="p">).</span><span class="nx">doc</span><span class="p">(</span><span class="nx">id</span><span class="p">);</span>
        <span class="k">await</span> <span class="nx">docRef</span><span class="p">.</span><span class="kd">set</span><span class="p">(</span><span class="nx">eventData</span><span class="p">,</span> <span class="p">{</span> <span class="na">merge</span><span class="p">:</span> <span class="kc">true</span> <span class="p">});</span>
        <span class="kd">const</span> <span class="nx">result</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">docRef</span><span class="p">.</span><span class="kd">get</span><span class="p">()</span>
        <span class="k">return</span> <span class="nx">result</span><span class="p">.</span><span class="nx">exists</span><span class="p">?</span> <span class="nx">result</span><span class="p">.</span><span class="nx">data</span><span class="p">():</span> <span class="p">{};</span> <span class="c1">// return the updated doc for later references</span>
    <span class="p">}</span> <span class="k">catch</span><span class="p">(</span><span class="nx">e</span><span class="p">)</span> <span class="p">{</span>
        <span class="nx">console</span><span class="p">.</span><span class="nx">error</span><span class="p">(</span><span class="nx">e</span><span class="p">);</span>
        <span class="k">throw</span> <span class="k">new</span> <span class="nb">Error</span><span class="p">(</span><span class="dl">"</span><span class="s2">Error saving event data!</span><span class="dl">"</span><span class="p">);</span>
    <span class="p">}</span>
<span class="p">};</span>
</code></pre></div></div>

<p>And since we have assumed that we have an available transit, we are going to publish a message to a pubsub topic to trigger the next event, <strong>bookings</strong>. First let’s create a topic for this purpose.</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gcloud pubsub topics create reservations.bookings
</code></pre></div></div>

<p>And please copy the output of that command and keep it saved, we are going to need it later. Same as before let’s have another environment varible for the topic name and wirte the function to publish the message. In the message we include <code class="language-plaintext highlighter-rouge">correlationId</code>, <code class="language-plaintext highlighter-rouge">numberOfSeats</code>, <code class="language-plaintext highlighter-rouge">transitId</code> and <code class="language-plaintext highlighter-rouge">userId</code>.</p>

<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">import</span> <span class="p">{</span> <span class="nx">PubSub</span> <span class="p">}</span> <span class="k">from</span> <span class="dl">"</span><span class="s2">@google-cloud/pubsub</span><span class="dl">"</span><span class="p">;</span>


<span class="kd">const</span> <span class="nx">BOOKINGS_TOPIC</span> <span class="o">=</span> <span class="nx">process</span><span class="p">.</span><span class="nx">env</span><span class="p">.</span><span class="nx">BOOKINGS_TOPIC</span> <span class="o">||</span> <span class="dl">"</span><span class="s2">reservations.bookings</span><span class="dl">"</span><span class="p">;</span>

<span class="k">export</span> <span class="kd">const</span> <span class="nx">publishMessage</span> <span class="o">=</span> <span class="k">async</span> <span class="p">(</span><span class="nx">topic</span><span class="p">,</span> <span class="nx">message</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="p">{</span>
    <span class="k">try</span> <span class="p">{</span>
        <span class="kd">const</span> <span class="nx">pubsubClient</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">PubSub</span><span class="p">();</span>
        <span class="kd">const</span> <span class="nx">dataBuffer</span> <span class="o">=</span> <span class="nx">Buffer</span><span class="p">.</span><span class="k">from</span><span class="p">(</span><span class="nx">JSON</span><span class="p">.</span><span class="nx">stringify</span><span class="p">(</span><span class="nx">message</span><span class="p">));</span>
        <span class="k">return</span> <span class="k">await</span> <span class="nx">pubsubClient</span><span class="p">.</span><span class="nx">topic</span><span class="p">(</span><span class="nx">topic</span><span class="p">)</span>
            <span class="p">.</span><span class="nx">publishMessage</span><span class="p">({</span> <span class="na">data</span><span class="p">:</span> <span class="nx">dataBuffer</span> <span class="p">});</span>
    <span class="p">}</span> <span class="k">catch</span> <span class="p">(</span><span class="nx">e</span><span class="p">)</span> <span class="p">{</span>
        <span class="nx">console</span><span class="p">.</span><span class="nx">error</span><span class="p">(</span><span class="nx">e</span><span class="p">);</span>
        <span class="k">throw</span> <span class="k">new</span> <span class="nb">Error</span><span class="p">(</span><span class="s2">`Error publishing message to </span><span class="p">${</span><span class="nx">topic</span><span class="p">}</span><span class="s2">!`</span><span class="p">);</span>
    <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>

<p>Now we have all helper functions and we can write the logic in the main funtion. Once the function is complted to deploy run,</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gostep deploy diff
</code></pre></div></div>

<h3 id="bookings-service">Bookings service</h3>

<p>The <strong>bookings</strong> function is responsible for hold the requested number of seats in selected transit until the global transaction is finished. The acting trigger of the function will be the <code class="language-plaintext highlighter-rouge">reservations.bookings</code> pubsub topic we create during the previous step. Once this service successfully locked the request number of seats it will publish a message to the relevenat pubsub topic to trigger the <strong>payments</strong> function.</p>

<p>Let’s start. To initialize the function run,</p>
<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gostep service init bookings version <span class="s2">"0.1.0"</span> <span class="nb">env </span>nodejs trigger pubsub
</code></pre></div></div>

<p>Now we have bootstrapped our cloud function in <code class="language-plaintext highlighter-rouge">{PROJECT_ROOT/src/bookings</code>. Let’s tell the funtion that it will triggered by the <code class="language-plaintext highlighter-rouge">reservations.bookings</code> topic. For that we can include the resource value we copied from the previous topic creation in the <code class="language-plaintext highlighter-rouge">{PROJECT_ROOT/src/bookings/function.json</code>.</p>

<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code>    <span class="dl">"</span><span class="s2">eventTrigger</span><span class="dl">"</span><span class="p">:</span> <span class="p">{</span>
        <span class="dl">"</span><span class="s2">eventType</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">providers/cloud.pubsub/eventTypes/topic.publish</span><span class="dl">"</span><span class="p">,</span>
        <span class="dl">"</span><span class="s2">resource</span><span class="dl">"</span><span class="p">:</span> <span class="dl">"</span><span class="s2">projects/{GCLOUD_PROJECT_ID}/topics/reservations.bookings</span><span class="dl">"</span>
    <span class="p">}</span>
</code></pre></div></div>

<p>Same as the <strong>reservations</strong> function, we need to save the local transaction’s event data with the <code class="language-plaintext highlighter-rouge">correlationId</code> and the <code class="language-plaintext highlighter-rouge">status</code> as <code class="language-plaintext highlighter-rouge">IN_PROGRESS</code> for later references. Also we can use same functions from the previous service to authorize requests to <strong>transits</strong> service and to publish the message to the next topic. What we can do is update the transit document to lock the requested number of seats.</p>

<div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">import</span> <span class="p">{</span> <span class="nx">GoogleAuth</span> <span class="p">}</span> <span class="k">from</span> <span class="dl">"</span><span class="s2">google-auth-library</span><span class="dl">"</span><span class="p">;</span>


<span class="kd">const</span> <span class="nx">TRANSITS_API</span> <span class="o">=</span> <span class="nx">process</span><span class="p">.</span><span class="nx">env</span><span class="p">.</span><span class="nx">TRANSITS_API</span> <span class="o">||</span> <span class="dl">"</span><span class="s2">{TRANSITS_API_HOST}/transits</span><span class="dl">"</span><span class="p">;</span>

<span class="k">export</span> <span class="kd">const</span> <span class="nx">getTransitsById</span> <span class="o">=</span> <span class="k">async</span> <span class="p">(</span><span class="nx">transitId</span><span class="p">,</span> <span class="nx">client</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="p">{</span>
    <span class="k">try</span> <span class="p">{</span>
        <span class="kd">const</span> <span class="nx">result</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">client</span><span class="p">.</span><span class="nx">request</span><span class="p">({</span>
            <span class="na">method</span><span class="p">:</span> <span class="dl">'</span><span class="s1">GET</span><span class="dl">'</span><span class="p">,</span>
            <span class="na">url</span><span class="p">:</span> <span class="s2">`</span><span class="p">${</span><span class="nx">TRANSITS_API</span><span class="p">}</span><span class="s2">?transitId=</span><span class="p">${</span><span class="nx">transitId</span><span class="p">}</span><span class="s2">`</span>
        <span class="p">});</span>
        <span class="k">return</span> <span class="nx">result</span><span class="p">.</span><span class="nx">length</span> <span class="o">&gt;</span> <span class="mi">0</span><span class="p">?</span> <span class="nx">result</span><span class="p">[</span><span class="mi">0</span><span class="p">]:</span> <span class="p">{};</span>
    <span class="p">}</span> <span class="k">catch</span> <span class="p">(</span><span class="nx">e</span><span class="p">)</span> <span class="p">{</span>
        <span class="nx">console</span><span class="p">.</span><span class="nx">error</span><span class="p">(</span><span class="nx">e</span><span class="p">);</span>
        <span class="k">throw</span> <span class="k">new</span> <span class="nb">Error</span><span class="p">(</span><span class="s2">`Error fetching transit data: </span><span class="p">${</span><span class="nx">transitId</span><span class="p">}</span><span class="s2">`</span><span class="p">);</span>
    <span class="p">}</span>
<span class="p">};</span>

<span class="k">export</span> <span class="kd">const</span> <span class="nx">updateTransitsById</span> <span class="o">=</span> <span class="k">async</span> <span class="p">(</span><span class="nx">id</span><span class="p">,</span> <span class="nx">newData</span><span class="p">,</span> <span class="nx">client</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="p">{</span>
    <span class="k">try</span><span class="p">{</span>
        <span class="k">return</span> <span class="k">await</span> <span class="nx">client</span><span class="p">.</span><span class="nx">request</span><span class="p">({</span>
            <span class="na">method</span><span class="p">:</span> <span class="dl">"</span><span class="s2">PUT</span><span class="dl">"</span><span class="p">,</span>
            <span class="na">url</span><span class="p">:</span> <span class="s2">`</span><span class="p">${</span><span class="nx">TRANSITS_API</span><span class="p">}</span><span class="s2">?transitId=</span><span class="p">${</span><span class="nx">id</span><span class="p">}</span><span class="s2">`</span><span class="p">,</span>
            <span class="na">body</span><span class="p">:</span> <span class="nx">newData</span>
        <span class="p">});</span>
    <span class="p">}</span> <span class="k">catch</span><span class="p">(</span><span class="nx">e</span><span class="p">)</span> <span class="p">{</span>
        <span class="nx">console</span><span class="p">.</span><span class="nx">error</span><span class="p">(</span><span class="nx">e</span><span class="p">);</span>
    <span class="p">}</span>
<span class="p">};</span>


<span class="k">export</span> <span class="kd">const</span> <span class="nx">main</span> <span class="o">=</span> <span class="k">async</span> <span class="p">(</span><span class="nx">eventData</span><span class="p">)</span> <span class="o">=&gt;</span> <span class="p">{</span>
    <span class="kd">const</span> <span class="nx">transactionData</span> <span class="o">=</span> <span class="nx">JSON</span><span class="p">.</span><span class="nx">parse</span><span class="p">(</span><span class="nx">atob</span><span class="p">(</span><span class="nx">eventData</span><span class="p">.</span><span class="nx">data</span><span class="p">));</span> <span class="c1">// Extract data from pubsub message</span>

    <span class="kd">const</span> <span class="nx">correlationId</span> <span class="o">=</span> <span class="nx">transactionData</span><span class="p">[</span><span class="dl">"</span><span class="s2">correlationId</span><span class="dl">"</span><span class="p">];</span>
    <span class="kd">const</span> <span class="nx">numberOfSeats</span> <span class="o">=</span> <span class="nb">Number</span><span class="p">(</span><span class="nx">transactionData</span><span class="p">[</span><span class="dl">"</span><span class="s2">numberOfSeats</span><span class="dl">"</span><span class="p">]);</span>
    <span class="kd">const</span> <span class="nx">transitId</span> <span class="o">=</span> <span class="nx">transactionData</span><span class="p">[</span><span class="dl">"</span><span class="s2">transitId</span><span class="dl">"</span><span class="p">];</span>
    <span class="kd">const</span> <span class="nx">userId</span> <span class="o">=</span> <span class="nx">transactionData</span><span class="p">[</span><span class="dl">"</span><span class="s2">userId</span><span class="dl">"</span><span class="p">];</span>

    <span class="kd">const</span> <span class="nx">auth</span> <span class="o">=</span> <span class="k">new</span> <span class="nx">GoogleAuth</span><span class="p">();</span>
    <span class="kd">const</span> <span class="nx">transitsApiClient</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">auth</span><span class="p">.</span><span class="nx">getIdTokenClient</span><span class="p">(</span><span class="nx">TRANSITS_API</span><span class="p">);</span>

    <span class="kd">const</span> <span class="nx">transit</span> <span class="o">=</span> <span class="k">await</span> <span class="nx">getTransitsById</span><span class="p">(</span><span class="nx">transitId</span><span class="p">,</span> <span class="nx">transitsApiClient</span><span class="p">);</span>
    <span class="k">await</span> <span class="nx">updateTransitsById</span><span class="p">(</span><span class="nx">transitId</span><span class="p">,</span> <span class="p">{</span>
                <span class="dl">"</span><span class="s2">lockedSeats</span><span class="dl">"</span><span class="p">:</span> <span class="nx">transit</span><span class="p">[</span><span class="dl">"</span><span class="s2">lockedSeats</span><span class="dl">"</span><span class="p">]</span> <span class="o">+</span> <span class="nx">numberOfSeats</span><span class="p">,</span>
                <span class="dl">"</span><span class="s2">availableSeats</span><span class="dl">"</span><span class="p">:</span> <span class="nx">transit</span><span class="p">[</span><span class="dl">"</span><span class="s2">availableSeats</span><span class="dl">"</span><span class="p">]</span> <span class="o">-</span> <span class="nx">numberOfSeats</span>
                <span class="p">},</span> <span class="nx">transitsApiClient</span><span class="p">);</span>
<span class="p">}</span>
</code></pre></div></div>

<p>And like before we will be creating the next pubsub topic to publish the message from <strong>bookings</strong>.</p>

<div class="language-sh highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gcloud pubsub topics create reservations.payments
</code></pre></div></div>

<p>After a successful seat locking, we will be publishing a message with <code class="language-plaintext highlighter-rouge">correlationId</code>, <code class="language-plaintext highlighter-rouge">numberOfClients</code> and <code class="language-plaintext highlighter-rouge">userId</code>.</p>

<p>Once the function has been completed we can deploy it using,</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gostep deploy diff
</code></pre></div></div>

<p>Great! Now we have covered common functionalities,</p>
<ul>
  <li>Consume HTTP requests</li>
  <li>Function to function direct communication(via HTTP)</li>
  <li>Read and update event data in firestore</li>
  <li>Publishing and subscribing to Pubsub topics</li>
</ul>

<p>This is more than enough for us to implement next services. Therefor afterwards, I will be explaining the function’s role only.</p>

<h3 id="payments-service">Payments service</h3>

<p>The payments service will consume the message from <code class="language-plaintext highlighter-rouge">reservations.payments</code> and publish a message to <code class="language-plaintext highlighter-rouge">reservations.bookingCompletions</code> or <code class="language-plaintext highlighter-rouge">reservations.bookingCancelletions</code> accordingly for a successful payment or for a failed payment.</p>

<h3 id="booking-completions-service">Booking completions service</h3>

<p>The booking completions will be consuming the messages from <code class="language-plaintext highlighter-rouge">reservations.bookingCompletions</code> topic, will be update the transit as the seat booking is completed and after that will update previously saved <strong>booking</strong> event’s status from <code class="language-plaintext highlighter-rouge">IN_PROGRESS</code> to <code class="language-plaintext highlighter-rouge">COMPLETED</code> for the relevant <code class="language-plaintext highlighter-rouge">correlationId</code>. Then the service will publish an message to the <code class="language-plaintext highlighter-rouge">reservations.reservationCompletions</code> topic.</p>

<h3 id="booking-cancellations-service">Booking cancellations service</h3>

<p>In the event of a payment failure, after consuming the message from the topic <code class="language-plaintext highlighter-rouge">reservations.bookingCancelletions</code> this function will rollback the locked seats in the relevant transit, will update <strong>booking</strong> event’s status from <code class="language-plaintext highlighter-rouge">IN_PROGRESS</code> to <code class="language-plaintext highlighter-rouge">FAILED</code> for the relevant <code class="language-plaintext highlighter-rouge">correlationId</code> and will pass the <code class="language-plaintext highlighter-rouge">correlationId</code> to the <code class="language-plaintext highlighter-rouge">reservations.reservationCancellations</code> topic.</p>

<h3 id="reservation-completions-service">Reservation completions service</h3>

<p>As the final step of a completed series of events the <strong>reservation completions service</strong> will consume the correlation id for the transaction from <code class="language-plaintext highlighter-rouge">reservations.reservationCompletions</code> and will update previously saved <code class="language-plaintext highlighter-rouge">reservations</code> event’s status from <code class="language-plaintext highlighter-rouge">IN_PROGRESS</code> to <code class="language-plaintext highlighter-rouge">COMPLETED</code>. After that a message will be published to the <code class="language-plaintext highlighter-rouge">reservations.notifications</code> topic to send the successful transaction notifications to the customer.</p>

<h3 id="reservation-cancellations-service">Reservation cancellations service</h3>

<p>Consuming the message from <code class="language-plaintext highlighter-rouge">reservations.reservationCancellations</code> this function will update previously saved <code class="language-plaintext highlighter-rouge">reservations</code> event’s status from <code class="language-plaintext highlighter-rouge">IN_PROGRESS</code> to <code class="language-plaintext highlighter-rouge">FAILED</code> and will publish a message to the <code class="language-plaintext highlighter-rouge">reservations.notifications</code> topic to send the failed transaction notifications status to the customer.</p>

<h2 id="securing-the-entrypoint">Securing the entrypoint</h2>

<p>After deploying all the services we can use Google API gateway to secure our <code class="language-plaintext highlighter-rouge">reservations</code> entrypoint of the transaction.
Please refer <a href="https://cloud.google.com/api-gateway/docs/secure-traffic-gcloud">API gatewey quickstart</a>.</p>

<p>🦖 Let’s look into <strong>Orchestration</strong> based solution in the next article.</p>]]></content><author><name>lpsandaruwan</name></author><category term="Posts" /><category term="Serverless" /><category term="gcp" /><category term="serverless" /><category term="saga" /><category term="choreography" /><category term="gostep" /><summary type="html"><![CDATA[Gostep: 👉 Guide Materials: 👉 Complete source code During the past few years, the microservices architecture(MSA) and serverless model have gained a lot of popularity in the industry. However, these technologies come with their own set of challenges. One substantial challenge is managing data in MSA due to its complexity. Considering common patterns for MSA data management we will be focusing on the Saga pattern in this article. The Saga pattern In order to manage business transactions across multiple microservices, the Saga pattern was introduced. Basically it is a series of local transactions; every transaction happens within the boundary of the micro-service, which every service will publish an event after the transaction for the next subsequent micro-service to perform the next transaction consuming the published event. This process will continue till the last transaction. In case any transaction failed in this series Saga will execute a series of fallback actions to undo the impact of all previous transactions. There are two approaches to implementing the Saga pattern. Choreography - The micro-service is responsible for emitting events eventually of its local transaction. The published event will trigger the execution of local transactions in microservices subscribed to the event. Also in this approach micro-service is responsible for handling the errors. Orchestration - A central orchestrator(a stateful coordinator) will trigger the local transactions in services and will maintain the global transaction status including handling errors. Now that we have a basic understanding of Saga pattern, we will discuss how to implement Saga pattern, defining an example for both approaches using Google Cloud Serverless model. The real world example Let’s consider a train ticket booking system. The workflow consists of, Send a seat reservation request Check for available transits in the database and proceed with seat booking. Hold the number of seats until payment is processed. Process the payment. Confirm the seat booking. Confirm the reservation and notify the customer. However if the system encountered any error while running a local transaction, the fallback sequence should be executed to undo all the changes happening in the global transaction to keep the ACID properties. Preparing the development environment (Please note that we won’t be using a real payment gateway or a notification service, beacause the main purpose of this article is to demonstrate how to use severless model for Saga.) To implement the solution we will be using Google Cloud serverless services, MongoDb and Javascript. Before we begin we must have, A billing enabled Google Cloud project Prior knowledge in Google Cloud Services Python, NodeJs, GCloud cli tools installed in your system(If you are using Windows, WSL might come in handy) Google Cloud CLI/Cloud console You can use both CLI tools or web console to create and modify services. In this article we will be mostly using CLI tools. Please follow https://cloud.google.com/sdk/docs/install to install the Google Cloud SDK. And once you installed the SDK run gcloud init command and follow instructions to configure credentials. Building the Cloud functions project structure To build the project structure and functions, we will be using gostep, a pythonic CLI tool that I created previously to manage implementations when there are a lot of cloud functions. To use gostep you need to have Subversion CLI, Python version 3 and Pip package manager installed(Setup a virtual environment of your own preference). When you are ready, run the command, pip install gostep. For more information please refer, http://lahirus.com/gostep-intro. Also please make sure that you have enabled Cloud build APIs(https://console.cloud.google.com/apis/library/cloudbuild.googleapis.com). Using gostep, let’s first create a Cloud Functions project. mkdir SagaChoreography &amp;&amp; cd SagaChoreography gostep auth init reservationsservice # Creates a service account credentials file gostep base init reservations location asia-east2 version "0.1.0" # Creates gostep project metadata files and directory structure. Now we can have the project base. Let’s move ahead with implementing local transactions and services. Choreography based solution For the demonstration we will be using, Pub/Sub for event sharing Firestore to store event data MongoDb as the transits service database Transits service This micro-service is responsible for CRUD operations on train entities. // Transit document schema { transitId: string, trainName: string, start: string, destination: string, day: string, departure: number, arrival: number, availableSeats: number, lockedSeats: number, totalSeats: number } As the database, we will be using MongoDB Atlas pay as you go service in the GCP marketplace After configuring the MongoDb instance, let’s create the transits function. gostep service init transits version "0.1.0" env nodejs This will create a boilerplate NodeJs cloud function in {PROJECT_ROOT}/src/transits and it can be executed as a http request after the deployment. Now let’s include the dependencies. cd src/transits npm install --save mongodb After creating the transits database and the collection, we can add MongoDb connection URI and collection name in the src/trains/functions.json as an environment variable. "environmentVariables": { "DB_URI": "mongodb+srv://&lt;username&gt;:&lt;password&gt;@&lt;your-cluster-url&gt;/&lt;dbname&gt;", "COLLECTION": "Transits" }, First let’s use these environment variables and create a function to connect to the database. import { MongoClient } from "mongodb"; const DB_URI = process.env.DB_URI || "&lt;Default DB con URI&gt;"; const dbClient = new MongoClient(DB_URI); const initDbClientConnection = async () =&gt; { try { await dbClient.connect(); } catch(e) { console.error(e); throw new Error("Database failed to connect!"); } }; And now let’s write 2 functions to find transits documents and save/update documents. const COLLECTION = process.env.COLLECTION || "Transits"; const query = async (queries) =&gt; { try { await initDbClientConnection(); const transits = dbClient.db().collection(COLLECTION); return await transits.find(queries).toArray(); } catch (e) { console.error(e); throw new Error("Failed to query transits!") } finally { await dbClient.close(); } } const save = async(transitId, patches) =&gt; { try { await initDbClientConnection(); const transits = dbClient.db().collection(COLLECTION); const targetData = { "$set": patches }; await transits.updateOne({ transitId: transitId }, targetData, { upsert: true }); } catch(e) { console.error(e); throw new Error("Failed to update transits!"); } finally { await dbClient.close(); } } In the main function we map GET and PUT https methods to above functions. export const main = async (req, res) =&gt; { if(req.method === "GET") { res.json(await query(req.query)); } else if(req.method === "PUT") { const transitId = req.query["transitId"]; if(!transitId) { res.status(400).send({ "error": "Invalid parameters!" }) } await save(transitId, req.body); res.status(201).send(); } else { res.status(400).json({ "error": "Invalid request" }); } } Great! Now we have our transits service. We can deploy it by running below command in the project root, gostep deploy diff After the deployment, transits service can be executed using http requests. But the endpoint is not available for the public. To test it locally, use the bearer token which you can obtain using the Gcloud cli. gcloud auth print-identity-token Reservations service Next, we are going to implement the entrypoint of the global transaction. Like before, let’s bootstrap a cloud function again. Run, gostep service init reservations version "0.1.0" env nodejs Now we have our boilerplate code in {PROJECT_ROOT}/src/reservations. Considering this scenario the reservations function is responsible for, Get the user request via a HTTP request. Call transits service and find out if there is any transit avaialable. If a transit is avialable publish an message to the relavent topic. Save the event data with it’s status as ‘IN_PROGRES’, to update later. We are going to keep the event data stored in a database. So that we can keep the status of the particular event to use later. For that purpose we use Google Cloud firestore(data store in native mode), which is a serverless easy to use document database. To enable Firestore run, gcloud firestore databases create --region=asia-southeast1 After that let’s install the dependencies. In the function root({PROJECT_ROOT}/src/resrevations) run, npm install --save "@google-cloud/firestore" "@google-cloud/pubsub" Let’s assume below payload as the request JSON. { "day": "Monday", "start": "Colombo", "destination": "Ragama", "numberOfSeats": 10, "userId": "xyz@gmail.com" } Once the user made his request we have to obtain the available transits for the requested day, start position and destination of the transit. To do that we will using a HTTP request to the transits service we implemeted before. Since the transits APIs are not publically available we have to use the google-auth-library to authorize requests from other services(See more). There is no need to add the auth library as a dependecy since it is an already included library in the cloud function runtime. First let’s add an environment variable for the transits API endpoint in {PROJECT_ROOT}/src/transits/function.json. "environmentVariables": { "TRANSITS_API": "{HOST_ADDRESS}/transits" } After that let’s authorize our request to fetch available transits. import { GoogleAuth } from "google-auth-library"; const TRANSITS_API = process.env.TRANSITS_API || "{DEFAULT_TRANSITS_HOST_ADDRESS}/transits"; export const getAvailableTransits = async (numberOfSeats, day, destination, start) =&gt; { try{ // Create an authorized client to invoke restricted Transits API. const auth = new GoogleAuth(); const transitsApiClient = await auth.getIdTokenClient(TRANSITS_API); const result = await transitsApiClient.request({ url: `${TRANSITS_API}?day=${day}&amp;destination=${destination}&amp;start=${start}`, method: "GET" }); return result.data.filter(element =&gt; element["availableSeats"] &gt;= numberOfSeats); } catch(e) { console.error(e); } }; Based on the result of the API request, we proceed further. Let’s assume that we got a list of available transits and we selected the topmost transit. Now we will be saving the event data in firestore, with a unique Id(a generated UUID as correlationId) as the global transation Id to identify the local transactions group and the status of the current event. It will aid to identify the local transaction for later references. Same as before we can add the firestore collection name(reffered as kind in firestore) of the event as an environment varibale in {PROJECT_ROOT}/src/transits/function.json. "environmentVariables": { "EVENT_DATA_COLLECTION": "reservations" } Now we can write our function to save event data in firestore. Please note that you don’t have to include configurations to authorize the connection to the firestore since the cloud function runtime has the authorized access to the firestore in the same project. import Firestore from "@google-cloud/firestore"; const EVENT_DATA_COLLECTION = process.env.EVENT_DATA_COLLECTION || "reservations"; export const saveEvent = async (id, eventData) =&gt; { try { const firestore = new Firestore(); const docRef = firestore.collection(EVENT_DATA_COLLECTION).doc(id); await docRef.set(eventData, { merge: true }); const result = await docRef.get() return result.exists? result.data(): {}; // return the updated doc for later references } catch(e) { console.error(e); throw new Error("Error saving event data!"); } }; And since we have assumed that we have an available transit, we are going to publish a message to a pubsub topic to trigger the next event, bookings. First let’s create a topic for this purpose. gcloud pubsub topics create reservations.bookings And please copy the output of that command and keep it saved, we are going to need it later. Same as before let’s have another environment varible for the topic name and wirte the function to publish the message. In the message we include correlationId, numberOfSeats, transitId and userId. import { PubSub } from "@google-cloud/pubsub"; const BOOKINGS_TOPIC = process.env.BOOKINGS_TOPIC || "reservations.bookings"; export const publishMessage = async (topic, message) =&gt; { try { const pubsubClient = new PubSub(); const dataBuffer = Buffer.from(JSON.stringify(message)); return await pubsubClient.topic(topic) .publishMessage({ data: dataBuffer }); } catch (e) { console.error(e); throw new Error(`Error publishing message to ${topic}!`); } } Now we have all helper functions and we can write the logic in the main funtion. Once the function is complted to deploy run, gostep deploy diff Bookings service The bookings function is responsible for hold the requested number of seats in selected transit until the global transaction is finished. The acting trigger of the function will be the reservations.bookings pubsub topic we create during the previous step. Once this service successfully locked the request number of seats it will publish a message to the relevenat pubsub topic to trigger the payments function. Let’s start. To initialize the function run, gostep service init bookings version "0.1.0" env nodejs trigger pubsub Now we have bootstrapped our cloud function in {PROJECT_ROOT/src/bookings. Let’s tell the funtion that it will triggered by the reservations.bookings topic. For that we can include the resource value we copied from the previous topic creation in the {PROJECT_ROOT/src/bookings/function.json. "eventTrigger": { "eventType": "providers/cloud.pubsub/eventTypes/topic.publish", "resource": "projects/{GCLOUD_PROJECT_ID}/topics/reservations.bookings" } Same as the reservations function, we need to save the local transaction’s event data with the correlationId and the status as IN_PROGRESS for later references. Also we can use same functions from the previous service to authorize requests to transits service and to publish the message to the next topic. What we can do is update the transit document to lock the requested number of seats. import { GoogleAuth } from "google-auth-library"; const TRANSITS_API = process.env.TRANSITS_API || "{TRANSITS_API_HOST}/transits"; export const getTransitsById = async (transitId, client) =&gt; { try { const result = await client.request({ method: 'GET', url: `${TRANSITS_API}?transitId=${transitId}` }); return result.length &gt; 0? result[0]: {}; } catch (e) { console.error(e); throw new Error(`Error fetching transit data: ${transitId}`); } }; export const updateTransitsById = async (id, newData, client) =&gt; { try{ return await client.request({ method: "PUT", url: `${TRANSITS_API}?transitId=${id}`, body: newData }); } catch(e) { console.error(e); } }; export const main = async (eventData) =&gt; { const transactionData = JSON.parse(atob(eventData.data)); // Extract data from pubsub message const correlationId = transactionData["correlationId"]; const numberOfSeats = Number(transactionData["numberOfSeats"]); const transitId = transactionData["transitId"]; const userId = transactionData["userId"]; const auth = new GoogleAuth(); const transitsApiClient = await auth.getIdTokenClient(TRANSITS_API); const transit = await getTransitsById(transitId, transitsApiClient); await updateTransitsById(transitId, { "lockedSeats": transit["lockedSeats"] + numberOfSeats, "availableSeats": transit["availableSeats"] - numberOfSeats }, transitsApiClient); } And like before we will be creating the next pubsub topic to publish the message from bookings. gcloud pubsub topics create reservations.payments After a successful seat locking, we will be publishing a message with correlationId, numberOfClients and userId. Once the function has been completed we can deploy it using, gostep deploy diff Great! Now we have covered common functionalities, Consume HTTP requests Function to function direct communication(via HTTP) Read and update event data in firestore Publishing and subscribing to Pubsub topics This is more than enough for us to implement next services. Therefor afterwards, I will be explaining the function’s role only. Payments service The payments service will consume the message from reservations.payments and publish a message to reservations.bookingCompletions or reservations.bookingCancelletions accordingly for a successful payment or for a failed payment. Booking completions service The booking completions will be consuming the messages from reservations.bookingCompletions topic, will be update the transit as the seat booking is completed and after that will update previously saved booking event’s status from IN_PROGRESS to COMPLETED for the relevant correlationId. Then the service will publish an message to the reservations.reservationCompletions topic. Booking cancellations service In the event of a payment failure, after consuming the message from the topic reservations.bookingCancelletions this function will rollback the locked seats in the relevant transit, will update booking event’s status from IN_PROGRESS to FAILED for the relevant correlationId and will pass the correlationId to the reservations.reservationCancellations topic. Reservation completions service As the final step of a completed series of events the reservation completions service will consume the correlation id for the transaction from reservations.reservationCompletions and will update previously saved reservations event’s status from IN_PROGRESS to COMPLETED. After that a message will be published to the reservations.notifications topic to send the successful transaction notifications to the customer. Reservation cancellations service Consuming the message from reservations.reservationCancellations this function will update previously saved reservations event’s status from IN_PROGRESS to FAILED and will publish a message to the reservations.notifications topic to send the failed transaction notifications status to the customer. Securing the entrypoint After deploying all the services we can use Google API gateway to secure our reservations entrypoint of the transaction. Please refer API gatewey quickstart. 🦖 Let’s look into Orchestration based solution in the next article.]]></summary></entry><entry><title type="html">Super charge a Google cloud functions project</title><link href="https://lpsandaruwan.github.io/posts/gostep-intro/" rel="alternate" type="text/html" title="Super charge a Google cloud functions project" /><published>2020-06-19T00:00:00+00:00</published><updated>2020-06-19T00:00:00+00:00</updated><id>https://lpsandaruwan.github.io/posts/gostep-intro</id><content type="html" xml:base="https://lpsandaruwan.github.io/posts/gostep-intro/"><![CDATA[<p><br />
When developing a microservices project with cloud functions, managing the cluster of functions all of them together can be a pain in the ass. That is why I thought of developing a simple cli tool to super charge the development and deployment process.</p>

<p>I named this little Pythonic tool as <strong>gostep</strong> a.k.a serverless templates provider for Google cloud platform. However this tool is still taking the baby steps. Hope to develop this to be more useful in future releases.</p>

<p>I would like to show you how it works up to now.</p>

<h3 id="first-of-all">First of all…</h3>
<p>You need to have installed below components to use gostep cli.</p>
<ol>
  <li>Python version 3.x with PyPI a.k.a pip(https://www.python.org/download/releases/3.0/, https://pypi.org/project/pip/)</li>
  <li>Gcloud sdk(https://cloud.google.com/sdk)</li>
  <li>subversion(https://subversion.apache.org)</li>
  <li>gostep(https://github.com/gostep-cli/gostep)</li>
</ol>

<h3 id="next-steps">Next steps…</h3>
<p>Now simply install gostep cli.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>pip install gostep
</code></pre></div></div>

<p>After installing gostep, using gcloud sdk log in to your google cloud platform account.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gcloud auth login
</code></pre></div></div>
<p>Once you logged in, select the gcloud project that you want to use for your serverless functions.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gcloud config set project {projectId}
</code></pre></div></div>
<p>Oh! wait, to list down project Ids <code class="language-plaintext highlighter-rouge">gcloud projects list</code> or <code class="language-plaintext highlighter-rouge">gostep gcloud projects</code> can be used.</p>

<h3 id="all-set">All set…</h3>
<p>Now we are ready build a cloud functions cluster.
First gostep needs a workspace directory, a gcloud service account and a credentials file for deployment purposes.
We can initiate them by this command, <code class="language-plaintext highlighter-rouge">gostep auth init {new_service_account_name} inside {workspace_directory}</code>.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gostep auth init my-service-account inside ./my-workspace
</code></pre></div></div>
<p>Now we can see a credentials file has been generated inside the workspace.
Next we need to create a configuration file which keeps the project skeleton. We need to chose a default region for that. Otherwise gostep will choose that for us. To get a list for our gcloud project <code class="language-plaintext highlighter-rouge">gostep gcloud locations</code> can be used. Now we can simply do,</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gostep base init {project_name} location{gcloud_region_id} version "0.1.0" explains {description} inside {workspace_directory}
</code></pre></div></div>
<p>In our case,</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>cd my-workspace
gostep base init my-new-project location asia-east2 version "0.1.0" explains "my sample project"
</code></pre></div></div>
<p>It’s geen light now to create cloud functions now. gostep has <a href="https://github.com/gostep-cli/gostep-templates">specified template structures</a> for this.
Let’s simply bootstrap a python cloud function. For this purpose,</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gostep service init {cloud_function_name} location {gcloud_region_id} version {service_version} env {runtime} explains {desciption} inside {workspace_directory}
</code></pre></div></div>
<p>Since we are in the workspace directory and we already set up a default location Id we won’t be using <code class="language-plaintext highlighter-rouge">location</code> and <code class="language-plaintext highlighter-rouge">inside</code> arguments.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gostep service init my-python-function version "0.1.0" env python
</code></pre></div></div>
<p>Let’s bootstrap another function with nodeJs.</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gostep service init my-nodejs-function version "0.1.0" env nodejs
</code></pre></div></div>
<p>We can see source files inside the <code class="language-plaintext highlighter-rouge">{workspace_directory}/src</code>. In our case inside, <code class="language-plaintext highlighter-rouge">my-workspace/src/my-nodejs-function</code> and <code class="language-plaintext highlighter-rouge">my-workspace/src/my-python-function</code>.</p>

<h3 id="great">Great…!</h3>
<p>Now our project is ready to get deployed.
Aight… Let’s do,</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gostep deploy diff
</code></pre></div></div>
<p><code class="language-plaintext highlighter-rouge">diff</code> keyword will only deploy the changes we made for our functions(gostep tracks md5 of the function directory). To deploy a single function it needs to be called by name. <code class="language-plaintext highlighter-rouge">gostep deploy {function_name}</code>. In our case,</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gostep deploy my-nodejs-function
</code></pre></div></div>

<h3 id="bravo">Bravo…!</h3>
<p>Now our functions are deployed and ready to be executed.
<img src="/assets/images/content/posts/gostep/functions.png" alt="cfunctions" /></p>

<h3 id="in-future-releases">In future releases…</h3>
<ul>
  <li>More templates, templates for go lang, templates for Spring framework, etc…</li>
  <li>Handle function triggers such as pubsub, events, etc…</li>
  <li>Run cloud functions cluster in local environment, so developers can benefit debugging.</li>
</ul>

<p>Please find the source code in https://github.com/gostep-cli/gostep.</p>]]></content><author><name>lpsandaruwan</name></author><category term="Projects" /><category term="DevOps" /><category term="gcp" /><category term="serverless" /><category term="gostep" /><category term="python" /><category term="cli" /><summary type="html"><![CDATA[When developing a microservices project with cloud functions, managing the cluster of functions all of them together can be a pain in the ass. That is why I thought of developing a simple cli tool to super charge the development and deployment process. I named this little Pythonic tool as gostep a.k.a serverless templates provider for Google cloud platform. However this tool is still taking the baby steps. Hope to develop this to be more useful in future releases. I would like to show you how it works up to now. First of all… You need to have installed below components to use gostep cli. Python version 3.x with PyPI a.k.a pip(https://www.python.org/download/releases/3.0/, https://pypi.org/project/pip/) Gcloud sdk(https://cloud.google.com/sdk) subversion(https://subversion.apache.org) gostep(https://github.com/gostep-cli/gostep) Next steps… Now simply install gostep cli. pip install gostep After installing gostep, using gcloud sdk log in to your google cloud platform account. gcloud auth login Once you logged in, select the gcloud project that you want to use for your serverless functions. gcloud config set project {projectId} Oh! wait, to list down project Ids gcloud projects list or gostep gcloud projects can be used. All set… Now we are ready build a cloud functions cluster. First gostep needs a workspace directory, a gcloud service account and a credentials file for deployment purposes. We can initiate them by this command, gostep auth init {new_service_account_name} inside {workspace_directory}. gostep auth init my-service-account inside ./my-workspace Now we can see a credentials file has been generated inside the workspace. Next we need to create a configuration file which keeps the project skeleton. We need to chose a default region for that. Otherwise gostep will choose that for us. To get a list for our gcloud project gostep gcloud locations can be used. Now we can simply do, gostep base init {project_name} location{gcloud_region_id} version "0.1.0" explains {description} inside {workspace_directory} In our case, cd my-workspace gostep base init my-new-project location asia-east2 version "0.1.0" explains "my sample project" It’s geen light now to create cloud functions now. gostep has specified template structures for this. Let’s simply bootstrap a python cloud function. For this purpose, gostep service init {cloud_function_name} location {gcloud_region_id} version {service_version} env {runtime} explains {desciption} inside {workspace_directory} Since we are in the workspace directory and we already set up a default location Id we won’t be using location and inside arguments. gostep service init my-python-function version "0.1.0" env python Let’s bootstrap another function with nodeJs. gostep service init my-nodejs-function version "0.1.0" env nodejs We can see source files inside the {workspace_directory}/src. In our case inside, my-workspace/src/my-nodejs-function and my-workspace/src/my-python-function. Great…! Now our project is ready to get deployed. Aight… Let’s do, gostep deploy diff diff keyword will only deploy the changes we made for our functions(gostep tracks md5 of the function directory). To deploy a single function it needs to be called by name. gostep deploy {function_name}. In our case, gostep deploy my-nodejs-function Bravo…! Now our functions are deployed and ready to be executed. In future releases… More templates, templates for go lang, templates for Spring framework, etc… Handle function triggers such as pubsub, events, etc… Run cloud functions cluster in local environment, so developers can benefit debugging. Please find the source code in https://github.com/gostep-cli/gostep.]]></summary></entry><entry><title type="html">Application Deployment in Apache Tomcat on GCE Using Ansible</title><link href="https://lpsandaruwan.github.io/posts/tomcat-gce-ansible-demo/" rel="alternate" type="text/html" title="Application Deployment in Apache Tomcat on GCE Using Ansible" /><published>2018-03-01T00:00:00+00:00</published><updated>2018-03-01T00:00:00+00:00</updated><id>https://lpsandaruwan.github.io/posts/tomcat-gce-ansible-demo</id><content type="html" xml:base="https://lpsandaruwan.github.io/posts/tomcat-gce-ansible-demo/"><![CDATA[<p>Think about a person who needs a cloud instance temporarily to deploy a web application to do tests frequently and throughout the time he deploys the application,
use it for a while and then deletes the instance to save the cost.
Or someone needs to create a cluster, thus he needs to instantiate several cloud servers at once,
install dependencies and deploy the application on each server. Doing these tasks by hand costs much effort and it is inefficient.
The way to make such scenarios easier, efficient and effective is making a reusable structure which does these repetitive tasks when we invoke it.
For that purpose, we use configuration management.</p>

<p><img src="/assets/images/content/posts/gce-tomcat-ansible/cm_showoff.png" alt="cm_what_happens" /></p>

<p>This tutorial is about a such scenario, to splash the easiness of using a reusable code base when deploying an application,
using <strong>Ansible</strong>(a radical and impressive configuration management tool with its capabilities and ease of use compared to other tools),
<strong>Google cloud platform</strong>(future potential cloud services with a good pricing model) and <strong>Apache Tomcat</strong>
(one of the most popular web servers in the Java community). For this purpose, we are going to use the <a href="https://github.com/lpsandaruwan/gce-tomcat-ansible-demo">gce-tomcat-ansible-demo</a> repository,
 a concatenation of Ansible playbooks(the reusable code base) implemented by me to reflect this task.
  Running this will create a Google compute engine in a given Google cloud platform project, install java,
   configure an Apache Tomcat server and will deploy a war file according to given metadata.
   (To understand playbooks knowing Ansible basics is more than enough. Refer <a href="http://docs.ansible.com/ansible/latest/intro_getting_started.html">Ansible documentation</a>)</p>

<p><img src="/assets/images/content/posts/gce-tomcat-ansible/playbook_flow.png" alt="playbook_flow" /></p>

<p><br /></p>
<h2 id="getting-started">Getting started</h2>

<p>To run the playbook you need to have <strong>a Google cloud platform account</strong>, <strong>a Google cloud platform project</strong> and <strong>a service account</strong>
to manipulate Google cloud project with appropriate roles and permission and <strong>an Ansible running machine</strong>.</p>

<p>There are several ways to install Ansible but here let’s install it using python pip package manager since the installation
is not going to depend on which operating system you use. But first make sure Python version 2,
Python development and Python pip(most probably python-dev and python-pip respectively,
refer <a href="https://packaging.python.org/guides/installing-using-linux-tools/#installing-pip-setuptools-wheel-with-linux-package-managers">installing pip with package managers for more information</a>) packages have been installed on the machine that you are
 going to install Ansible. To make sure, try running the command below.</p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash">pip list</code></pre></figure>

<p><img src="/assets/images/content/posts/gce-tomcat-ansible/pip_list.png" alt="pip_list" /></p>

<p>And then clone the repository <a href="https://github.com/lpsandaruwan/gce-tomcat-ansible-demo">gce-tomcat-ansible-demo</a>, which contains playbooks to create a Google compute engine instance,
 install Java, configure a Apache Tomcat server and deploy a given war file on the configured application server.</p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash">git clone https://github.com/lpsandaruwan/gce-tomcat-ansible-demo.git</code></pre></figure>

<p>Then change the current working directory into the cloned repository.</p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">cd </span>gce-tomcat-ansible-demo</code></pre></figure>

<p>Now use the <code class="language-plaintext highlighter-rouge">requirements.txt</code> to install appropriate Python pip package versions which this playbook has been written and tested for.</p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash">pip <span class="nb">install</span> <span class="nt">-r</span> requirements.txt</code></pre></figure>

<p>This will install ansible and apache-libcloud(a fine interface to deal with popular cloud services)
Python packages which we are going to use for manipulating Google cloud project.</p>

<p>Make sure that you have a working Google cloud platform project(if not refer <a href="https://cloud.google.com/resource-manager/docs/creating-managing-projects">creating and manage projects</a>),
and a service account assigned to it(do not use the default service account since it has full permission over the project,
refer <a href="https://cloud.google.com/compute/docs/access/service-accounts">service accounts</a> for more information) and then obtain the project ID,
private JSON keyfile(you can obtain the JSON key file when creating a new service account,
if not refer <a href="https://cloud.google.com/storage/docs/authentication#service-account-credentials">service account credentials</a>) and the service account email from them.
Now we have to configure Ansible running machine to access the GCP project.
Here Let’s use Google cloud SDK to make things easier(refer <a href="https://cloud.google.com/sdk/downloads#versioned">install Google cloud SDK</a>).
After installing the SDK run the below command to initialize.</p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash">gcloud init</code></pre></figure>

<p><img src="/assets/images/content/posts/gce-tomcat-ansible/gcloud_init.png" alt="gcloud_init" /></p>

<p>And it will direct you to the web page in your browser. From there allow the access to the SDK.
 Now in the terminal select the appropriate project ID. After that you should be able to run playbooks on the appropriate project and manipulate it.
  If you run into a permission problem connecting the instance configure SSH authentication using cloud SDK tools by running the command below.
  (refer <a href="https://cloud.google.com/sdk/gcloud/reference/compute/config-ssh">gcloud compute config-ssh</a>)</p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash">gcloud compute config-ssh</code></pre></figure>

<p><br /></p>

<h2 id="play-it">Play it</h2>

<p>Now configure the file <code class="language-plaintext highlighter-rouge">gce-vars/authentication</code> and update the obtained metadata from GCP project and service account in playbook.</p>

<figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="na">project_id</span><span class="pi">:</span> <span class="s">“project-id-193706”</span>
<span class="na">credentials_file</span><span class="pi">:</span> <span class="s">“/path/to/private/json/key/file”</span>
<span class="na">service_account_email</span><span class="pi">:</span> <span class="s">“tomcat-ansible-demo@service-account-193706.iam.gserviceaccount.com”</span></code></pre></figure>

<p>After that change instance metadata in <code class="language-plaintext highlighter-rouge">gce-vars/instance</code> as you need. Here,
we are going to add firewall rules to allow HTTP traffic on 8080 ports for Apache Tomcat server</p>

<figure class="highlight"><pre><code class="language-yaml" data-lang="yaml"><span class="na">name</span><span class="pi">:</span> <span class="s">tomcat-ansible-demo</span>
<span class="na">type</span><span class="pi">:</span> <span class="s">f1-micro</span>
<span class="na">image</span><span class="pi">:</span> <span class="s">debian-9</span>
<span class="na">zone</span><span class="pi">:</span> <span class="s">europe-west1-b</span>
<span class="na">allowed_ports_tcp</span><span class="pi">:</span> <span class="s">tcp:8080</span>
<span class="na">allowed_ports_udp</span><span class="pi">:</span> <span class="s">udp:8080</span></code></pre></figure>

<p>Then set the <code class="language-plaintext highlighter-rouge">ANSIBLE_HOSTS</code> environment variable required by Ansible for SSH interactions.
To do that simply put hostnames in <code class="language-plaintext highlighter-rouge">hosts</code> file and import it as below.</p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">export </span><span class="nv">ANSIBLE_HOSTS</span><span class="o">=</span>hosts</code></pre></figure>

<p>Now you should be able to run this project. Simply run the main playbook.</p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash">ansible-playbook run.ym</code></pre></figure>

<p>Final output will be as below. And after a successful run you will have your application deployed in an
Apache Tomcat server on a Google compute engine instance.</p>

<p><img src="/assets/images/content/posts/gce-tomcat-ansible/final.png" alt="final" />
<img src="/assets/images/content/posts/gce-tomcat-ansible/web.png" alt="web" />
<br /></p>

<h2 id="appendix">Appendix</h2>

<p>(If you wish to change Java version, Tomcat version etc.
configure main.yml in defaults directory in roles. They contain configuration variables with lower priorities.)</p>

<figure class="highlight"><pre><code class="language-text" data-lang="text">gce-tomcat-ansible-demo
|------gce_vars				# variables related to Google cloud platform
|	|	authentication		# Google project and service account related metadata
|	|	instance		# GCE instance related metadata
|
|-------roles
|	|-------java				# role to install Java
|	|	|-------defaults
|	|	|	|	main.yml	# default variables for java role
|	|	|
|	|	|-------tasks
|	|	|	|	main.yml	# tasks to download and install Java
|	|
|	|-------tomcat
|	|	|-------defaults
|	|	|	|	main.yml	# default variables for tomcat role
|	|	|
|	|	|-------files
|	|	|	|	tomcat-users.xml	# set tomcat manager credentials
|	|	|
|	|	|-------tasks
|	|	|	|	main.yml	# tasks to download and configure tomcat
|	|
|	|-------tomcat-deploy
|	|	|-------defaults
|	|	|	|	main.yml	# default variables for tomcat-deploy role
|	|	|
|	|	|-------tasks
|	|	|	|	main.yml	# tasks to deploy the given war file
|
|	hosts					# ansible hosts
|	bootstrap-instance.yml			# playbook to initiate google cloud instance
|	deploy-war.yml				# playbook to deploy war file
|	install-java.yml			# playbook to install Java
|	install-tomcat.yml			# playbook to install Apache Tomcat
|	run.yml					# main playbook to run</code></pre></figure>

<p><br />
<a rel="license" href="http://creativecommons.org/licenses/by/4.0/"><img alt="Creative Commons License" style="border-width:0" src="https://i.creativecommons.org/l/by/4.0/88x31.png" /></a><br /><span xmlns:dct="http://purl.org/dc/terms/" href="http://purl.org/dc/dcmitype/Text" property="dct:title" rel="dct:type">gce-tomcat-ansible-demo post</span> by <a xmlns:cc="http://creativecommons.org/ns#" href="http://lahiru.site/blog/2018/tomcat-gce-ansible-demo/" property="cc:attributionName" rel="cc:attributionURL">Lahiru Pathirage</a> is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by/4.0/">Creative Commons Attribution 4.0 International License</a>.<br />Based on a work at <a xmlns:dct="http://purl.org/dc/terms/" href="https://github.com/lpsandaruwan/gce-tomcat-ansible-demo" rel="dct:source">https://github.com/lpsandaruwan/gce-tomcat-ansible-demo</a>.</p>]]></content><author><name>lpsandaruwan</name></author><category term="Posts" /><category term="DevOps" /><category term="tomcat" /><category term="gce" /><category term="ansible" /><category term="configuration-management" /><summary type="html"><![CDATA[Think about a person who needs a cloud instance temporarily to deploy a web application to do tests frequently and throughout the time he deploys the application, use it for a while and then deletes the instance to save the cost. Or someone needs to create a cluster, thus he needs to instantiate several cloud servers at once, install dependencies and deploy the application on each server. Doing these tasks by hand costs much effort and it is inefficient. The way to make such scenarios easier, efficient and effective is making a reusable structure which does these repetitive tasks when we invoke it. For that purpose, we use configuration management.]]></summary></entry><entry><title type="html">Setting up Creative Labs USB DAC volume knob on Linux</title><link href="https://lpsandaruwan.github.io/posts/sb1095-volume-knob-linux/" rel="alternate" type="text/html" title="Setting up Creative Labs USB DAC volume knob on Linux" /><published>2017-05-11T00:00:00+00:00</published><updated>2017-05-11T00:00:00+00:00</updated><id>https://lpsandaruwan.github.io/posts/sb1095-volume-knob-linux</id><content type="html" xml:base="https://lpsandaruwan.github.io/posts/sb1095-volume-knob-linux/"><![CDATA[<p>Lately I bought a <a href="http://us.creative.com/p/sound-blaster/sound-blaster-x-fi-surround-5-1-pro">Creative Labs SB1095</a>, a 5.1 USB DAC for my laptop.
This USB sound card works perfectly on Linux.
The sound quality is better than the integrated,
but it has a volume knob on it and a remote controller,
which does not support out of the box by Linux distributions which I have tried(Ubuntu, Linux Mint, OpenSuse, Arch Linux).
After digging up the internet a little bit I perceived that it is required to configure a software volume controller to handle the volume knob.
However, I did not want to go for that kind of advanced configurations, and finally found an easy workaround for this purpose. Would like to take down the steps,
so it might help others which have the same issue.</p>

<p>To achieve this I used lirc (an application which interprets IR actions) to detect volume knob and remote actions and wrapped it with irexec (trigger actions for lirc inputs) to change system sound volume.</p>

<p>I am currently using Linux Mint 18.1, so this solution will work perfectly with Ubuntu 16.04 derivatives.
For other Linux distributions please follow <a href="https://sites.google.com/site/klaasdc/runeaudio-creative-xfi-5-1-usb">this guide</a>.
<br /><br /></p>

<h4 id="requirements">Requirements</h4>
<p>First I installed these packages, selected Creative USB IR Receiver (SB0540) for remote controller configuration(not the SB1095,
but it has the same configurations) and selected none for IR transmitter in appearing configuration menus when installing.</p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">sudo </span>apt <span class="nb">install </span>lirc lirc-x</code></pre></figure>

<p><br /></p>

<h4 id="configurations">Configurations</h4>
<p>Then I changed the <code class="language-plaintext highlighter-rouge">REMOTE_DRIVER</code> in lirc hardware configuration file <code class="language-plaintext highlighter-rouge">/etc/lirc/hardware.conf</code>, left other settings unchanged.</p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># ~/.lircrc</span>
begin
 remote <span class="o">=</span> <span class="k">*</span>
 prog <span class="o">=</span> irexec
 config <span class="o">=</span> amixer <span class="nt">-D</span> pulse sset Master 5%-
 button <span class="o">=</span> vol-
 repeat <span class="o">=</span> 1
end

begin
 remote <span class="o">=</span> <span class="k">*</span>
 prog <span class="o">=</span> irexec
 config <span class="o">=</span> amixer <span class="nt">-D</span> pulse sset Master 5%+
 button <span class="o">=</span> vol+
 repeat <span class="o">=</span> 1
end

begin
 remote <span class="o">=</span> <span class="k">*</span>
 prog <span class="o">=</span> irexec
 config <span class="o">=</span> amixer <span class="nt">-D</span> pulse sset Master 0
 button <span class="o">=</span> mute
end</code></pre></figure>

<p><br /></p>

<h4 id="test">Test</h4>
<p>Then I restarted lirc daemon and loaded irexec daemon.</p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">sudo </span>service lirc restart
irexec <span class="nt">-d</span> <span class="c"># make a startup entry to load on system boot up</span></code></pre></figure>

<p>Now I have the volume knob working just fine.
<br />If the volume is not changing check whether lirc detects inputs by the USB device using the command <code class="language-plaintext highlighter-rouge">irw</code></p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="c"># irw sample output for volume knob changes</span>
0000000000000010 01 vol+ RM-1500
0000000000000010 00 vol+ RM-1500
0000000000000010 01 vol+ RM-1500
000000000000000f 03 vol- RM-1500
000000000000000f 00 vol- RM-1500
000000000000000d 00 mute RM-1500</code></pre></figure>

<p>If the output is something like the above, try changing config values(commands to change sound volume) in the <code class="language-plaintext highlighter-rouge">~/.lircrc</code> after that try restarting lirc daemon and loading <code class="language-plaintext highlighter-rouge">irexec</code> again.
<br /></p>

<h5 id="sources">Sources</h5>
<p><a href="http://alsa.opensrc.org/Usb-audio#Creative_USB_X-Fi_Surround_5.1">http://alsa.opensrc.org/Usb-audio#Creative_USB_X-Fi_Surround_5.1</a>
<br /><a href="http://www.lirc.org/html/configure.html">http://www.lirc.org/html/configure.html</a>
<br /><br /><br /></p>]]></content><author><name>lpsandaruwan</name></author><category term="Posts" /><category term="Linux" /><category term="usb dac" /><category term="creative labs" /><category term="sb1095" /><category term="linux" /><category term="volume knob" /><summary type="html"><![CDATA[Lately I bought a Creative Labs SB1095, a 5.1 USB DAC for my laptop. This USB sound card works perfectly on Linux. The sound quality is better than the integrated, but it has a volume knob on it and a remote controller, which does not support out of the box by Linux distributions which I have tried(Ubuntu, Linux Mint, OpenSuse, Arch Linux). After digging up the internet a little bit I perceived that it is required to configure a software volume controller to handle the volume knob. However, I did not want to go for that kind of advanced configurations, and finally found an easy workaround for this purpose. Would like to take down the steps, so it might help others which have the same issue.]]></summary></entry><entry><title type="html">Continuous Code Quality On My OpenSource Project</title><link href="https://lpsandaruwan.github.io/posts/github-sonarqube/" rel="alternate" type="text/html" title="Continuous Code Quality On My OpenSource Project" /><published>2017-04-20T00:00:00+00:00</published><updated>2017-04-20T00:00:00+00:00</updated><id>https://lpsandaruwan.github.io/posts/github-sonarqube</id><content type="html" xml:base="https://lpsandaruwan.github.io/posts/github-sonarqube/"><![CDATA[<p>Good quality in code plays an essential role when it comes to software,
thus it assets efficiency, reliability, robustness, portability, maintainability and readability like essential factors.
Considering a GitHub project, there are plenty of options to measure code quality.
Considering options I would like to chose <a href="https://www.sonarqube.org/">SonarQube</a> for this particular purpose.
Let me take down the steps, how I used SonarQube to measure code quality using a Java project, one of my GitHub hosted projects, <a href="https://github.com/lpsandaruwan/depli">Depli</a>.</p>

<p>NB: The best way to analyze a maven project is to use the maven sonar plugin <a href="https://docs.sonarqube.org/display/SCAN/Analyzing+with+SonarQube+Scanner+for+Maven">as the SonarQube docs says</a>.
You do not require a <code class="language-plaintext highlighter-rouge">sonar-project.properties</code> in that case.
<br /><br /></p>

<h4 id="step-1---create-an-account-in-travis-ciorg">Step 1 - Create an account in travis-ci.org</h4>

<p>SonarQube needs <a href="https://docs.sonarqube.org/display/SONAR/Analyzing+Source+Code">sonar-runner</a> to analyze the code.
To run the analysis process using sonar-runner on code changes continuously, the ideal solution is using a CI server.
Here I had to use <a href="https://travis-ci.org/">Travis-CI</a> since it is the perfect matured CI solution for GitHub projects.</p>

<p>I created and logged into <a href="https://travis-ci.org/">Travis</a> using my GitHub account and <a href="https://travis-ci.org/getting_started">activated it</a> for my repository.
<br /><br /></p>

<h4 id="step-2---create-an-account-in-sonarqubecom">Step 2 - Create an account in sonarqube.com</h4>

<p>Then I created and logged into <a href="https://sonarqube.com">SonarQube</a> using the same GitHub account.
<br /><br /></p>

<h4 id="step-3---create-travis-ci-configuration-file">Step 3 - Create Travis-CI configuration file</h4>

<p>Next step was to create a configuration file for Travis-CI to instruct it to how to run the sonar-scanner as known as sonar-runner.
To do that I created a <code class="language-plaintext highlighter-rouge">.travis.yml</code>, A YAML file in my project’s root directory.</p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash">dist: trusty <span class="c"># chose ubuntu trusty as the worker</span>

<span class="nb">sudo</span>: required

addons:
  sonarqube:
    organization: lpsandaruwan-github <span class="c"># organization token from https://sonarqube.com/account/organizations</span>

jdk:
- oraclejdk8

script:
- mvn clean <span class="nb">install</span> <span class="nt">-DskipTests</span> <span class="c"># skipped tests because I have not written.</span>
- sonar-scanner <span class="c"># tell travis to run sonar scanner</span>

cache:
  directories:
  - <span class="s2">"</span><span class="nv">$HOME</span><span class="s2">/.sonar/cache"</span></code></pre></figure>

<p><br /></p>

<h4 id="step-4---create-a-sonarqube-token">step 4 - Create a SonarQube token</h4>

<p>After creating a Travis configuration file I generated a <a href="https://docs.sonarqube.org/display/SONAR/User+Token">security token</a> and copied it to clipboard,
for Travis to use when updating SonarQube database.
<br /><br /></p>

<h4 id="step-5---encrypt-sonarqube-token">Step 5 - Encrypt SonarQube token</h4>
<p>Public access to a security token is a bad thing. So I had to encrypt the SonarQube token when inserting it to Travis configuration file.
To achieve that I used <a href="https://rubygems.org/gems/travis"><code class="language-plaintext highlighter-rouge">travis</code></a> from ruby gems.</p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="nb">cd</span> /path/to/my/project/root
travis encrypt MY_SONARQUBE_TOKEN <span class="nt">--add</span> addons.sonarqube.token</code></pre></figure>

<p><br /></p>

<h4 id="step-6---create-a-sonarqube-configuration-file">Step 6 - Create a SonarQube configuration file</h4>
<p>A metadata file including project details is required for SonarQube. So I created <code class="language-plaintext highlighter-rouge">sonar-project.properties</code> in my project’s root.
Here <code class="language-plaintext highlighter-rouge">sonar.sources</code> is the place where sonar-scanner starts to analyze.</p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash">sonar.projectKey<span class="o">=</span>com.sonarqube.lpsandaruwan.depli
sonar.projectName<span class="o">=</span>Depli - JVM Monitoring Dashboard
sonar.projectVersion<span class="o">=</span>0.2.0-SNAPSHOT

sonar.links.homepage<span class="o">=</span>https://lahirus.com/depli
sonar.links.ci<span class="o">=</span>https://travis-ci.org/lpsandaruwan/depli
sonar.links.scm<span class="o">=</span>https://github.com/lpsandaruwan/depli
sonar.links.issue<span class="o">=</span>https://github.com/lpsandaruwan/depli/issues

sonar.sources<span class="o">=</span>src/main</code></pre></figure>

<p><br /></p>

<h4 id="step-7---add-status-to-read-me">Step 7 - Add status to read me</h4>
<p>Now to display the project status on readme, I added labels from Travis and SonarQube on <code class="language-plaintext highlighter-rouge">README.md</code> file.</p>

<figure class="highlight"><pre><code class="language-bash" data-lang="bash"><span class="o">[![</span>Build Status]<span class="o">(</span>https://travis-ci.org/USERNAME/PROJECT_NAME.png<span class="o">)](</span>https://travis-ci.org/USERNAME/PROJECT_NAME<span class="o">)</span>
<span class="o">[![</span>Quality Gate]<span class="o">(</span>https://sonarqube.com/api/badges/gate?key<span class="o">=</span>SONAR_PROJECT_KEY<span class="o">)](</span>https://sonarqube.com/dashboard/index/SONAR_PROJECT_KEY<span class="o">)</span></code></pre></figure>

<p><br /></p>

<h3 id="bravo">Bravo!</h3>
<p>After following above steps I pushed all changes to GitHub. And waited until Travis sent me a mail confirming that my build has been successful.
Now the readme is displaying the build status and whether my project has passed the quality gate.</p>

<p><img src="/assets/images/content/posts/github-sonarqube/readme.png" alt="readme" />
<br /><br /></p>

<p>By clicking on the <code class="language-plaintext highlighter-rouge">quality gate</code> badge, I can access the SonarQube dashboard, detailed analysis of code quality of my repository.</p>

<p><img src="/assets/images/content/posts/github-sonarqube/sonarqube_page.png" alt="readme" />
<br /><br /></p>

<p>Please refer my open source project, <a href="https://github.com/lpsandaruwan/depli">https://github.com/lpsandaruwan/depli</a> if there is any doubt.
<br /><br /></p>]]></content><author><name>lpsandaruwan</name></author><category term="Posts" /><category term="DevOps" /><category term="code-quality" /><category term="github" /><category term="travis-ci" /><category term="sonarqube" /><summary type="html"><![CDATA[Good quality in code plays an essential role when it comes to software, thus it assets efficiency, reliability, robustness, portability, maintainability and readability like essential factors. Considering a GitHub project, there are plenty of options to measure code quality. Considering options I would like to chose SonarQube for this particular purpose. Let me take down the steps, how I used SonarQube to measure code quality using a Java project, one of my GitHub hosted projects, Depli.]]></summary></entry><entry><title type="html">Continuous listening to remote text files using python</title><link href="https://lpsandaruwan.github.io/posts/log-tracker/" rel="alternate" type="text/html" title="Continuous listening to remote text files using python" /><published>2017-04-14T00:00:00+00:00</published><updated>2017-04-14T00:00:00+00:00</updated><id>https://lpsandaruwan.github.io/posts/log-tracker</id><content type="html" xml:base="https://lpsandaruwan.github.io/posts/log-tracker/"><![CDATA[<p>Log Tracker is a simple wrapper around Python <strong>paramiko</strong> to track text files using SSH.
It gives you the ability to create custom python functions to track and analyze log files the way you want.
A user can access the contents in multiple log files at the same time also.
Using a custom function a user can display a log content in a web interface using Flask like lightweight web service,
so then anyone can analyze contents easily, without wasting time to login into servers and download contents.
For more information please refer below links.
<br /><br /></p>

<p>Website: <a href="https://lahirus.com/log-tracker">https://lahirus.com/log-tracker</a></p>

<p>Wiki: <a href="https://github.com/lpsandaruwan/log-tracker/wiki">https://github.com/lpsandaruwan/log-tracker/wiki</a></p>

<p>License: <a href="https://github.com/lpsandaruwan/log-tracker/blob/master/README.md">GPLv3 or later</a></p>

<p>Source code: <a href="https://github.com/lpsandaruwan/log-tracker">https://github.com/lpsandaruwan/log-tracker</a></p>

<p>Releases: <a href="https://github.com/lpsandaruwan/log-tracker/releases">https://github.com/lpsandaruwan/log-tracker/releases</a></p>]]></content><author><name>lpsandaruwan</name></author><category term="[&quot;Projects&quot;, &quot;DevOps&quot;]" /><category term="python" /><category term="log files" /><category term="paramiko" /><category term="wrapper application" /><summary type="html"><![CDATA[Log Tracker is a simple wrapper around Python paramiko to track text files using SSH. It gives you the ability to create custom python functions to track and analyze log files the way you want. A user can access the contents in multiple log files at the same time also. Using a custom function a user can display a log content in a web interface using Flask like lightweight web service, so then anyone can analyze contents easily, without wasting time to login into servers and download contents. For more information please refer below links.]]></summary></entry><entry><title type="html">Depli - A JVM monitor application</title><link href="https://lpsandaruwan.github.io/posts/depli/" rel="alternate" type="text/html" title="Depli - A JVM monitor application" /><published>2017-04-03T00:00:00+00:00</published><updated>2017-04-03T00:00:00+00:00</updated><id>https://lpsandaruwan.github.io/posts/depli</id><content type="html" xml:base="https://lpsandaruwan.github.io/posts/depli/"><![CDATA[<p>Depli provides you the 10-second solution for monitoring JVMs. Just add a JMX remote connection using webUI and see how it works.
Depli provides you a rich UI, you can even search for running threads, classpaths etc.
This handsome tool has been released under GPL license on GitHub.</p>

<p><img src="/assets/images/content/depli.png" alt="depli" /></p>

<p>Website: <a href="https://lahirus.com/depli">https://lahirus.com/depli</a></p>

<p>Wiki: <a href="https://github.com/lpsandaruwan/depli/wiki">https://github.com/lpsandaruwan/depli/wiki</a></p>

<p>License: <a href="https://github.com/lpsandaruwan/depli/blob/master/README.md">GPLv3 or later</a></p>

<p>Source code: <a href="https://github.com/lpsandaruwan/depli">https://github.com/lpsandaruwan/depli</a></p>

<p>Releases: <a href="https://github.com/lpsandaruwan/depli/releases">https://github.com/lpsandaruwan/depli/releases</a></p>]]></content><author><name>lpsandaruwan</name></author><category term="Projects" /><category term="DevOps" /><category term="java" /><category term="spring-boot" /><category term="angularjs" /><category term="jvm performance" /><summary type="html"><![CDATA[Depli provides you the 10-second solution for monitoring JVMs. Just add a JMX remote connection using webUI and see how it works. Depli provides you a rich UI, you can even search for running threads, classpaths etc. This handsome tool has been released under GPL license on GitHub.]]></summary></entry></feed>