Skip to main content

Integrating Gemini with mcp.run on Android

This tutorial guides you through connecting an Android application with mcp.run's tool ecosystem. You'll learn how to create a chat interface that can interact with external tools and APIs through mcp.run.

Google now provides access to their flagship Gemini models.

info

Find complete source code for this example in the mcpx4j repository examples directory.

For a video walkthrough:

warning

Android support in MCPX4J is still experimental! Gemini support for function calling is also a beta feature.

Prerequisites​

Before starting, ensure you have the following:

Additionally, in order to use Gemini:

Finally, optionally, for the servlets:

Setting up mcp.run​

You'll need an mcp.run account and session ID. Here's how to get started:

  1. Run this command in your terminal: 
    npx --yes -p @dylibso/mcpx@latest gen-session
  2. Your browser will open to complete authentication through GitHub
  3. After authenticating, return to your terminal and save the provided session key
Security Note

Keep your mcp.run session ID and OpenAI API key secure. Never commit these credentials to code repositories or expose them publicly.

Required Tools​

This tutorial requires three mcp.run servlets:

Install all servlets by:

  1. Visiting each URL
  2. Clicking the Install button
  3. Verifying they appear in your install profile

Project Setup​

Create a new Android project:

  • Open Android Studio
  • Create a new Project for "Phone and Tablet"
  • Choose the Gemini API Starter
  • In the following window configure
    • the application name (in our case "Mcpx4jAndroid")
    • the package name com.dylibso.mcpx4j.examples.android
    • the minimum SDK (in this example we will use API 35)
    • the Kotlin configuration language for Gradle

You will be asked for your Gemini API Key.

Now you can already run the project. It will look like this:

Gemini Android Demo Initial

At this point, you Gemini API key has been written in the /local.properties at the root of the project.

You can use this file for other secret values, it will not be committed to your repository. Your /local.properties should then look like this:

sdk.dir=<path to your local Android SDK>
apiKey=<Your Gemini API Key>
mcpRunKey=<Your MCP.RUN session key>

We will also need to add a few more dependencies. Open /gradle/libs.versions.toml:

[libraries]
...
mcpx4j = { group = "com.github.dylibso.mcpx4j", name = "core", version.ref = "mcpx4j" }
jackson-databind = { group = "com.fasterxml.jackson.core", name = "jackson-databind", version.ref = "jackson" }

Then, let's add the version number to the [versions] section. Make also sure that you are using a recent version of the Generative AI SDK.

[versions]
...
generativeai = "0.9.0"
mcpx4j = ...
jackson = ...

Then, let's refer them in /app/build.gradle.kts:

dependencies {
    ...
    implementation(libs.mcpx4j)
    implementation(libs.jackson.databind)
}

Finally, MCPX4J is currently available on JitPack. Let us also configre the JitPack repo in /settings.gradle.kts:

dependencyResolutionManagement {
    repositoriesMode.set(RepositoriesMode.FAIL_ON_PROJECT_REPOS)
    repositories {
        google()
        mavenLocal()
        maven { url = uri("https://jitpack.io") }
        mavenCentral()
    }
}

Creating the mcp.run Function Repository​

In mcp.run, Tools are organized in Servlets. Every Servlet may contain several tools. Tools and servlets are exposed by mcpx4j as McpxTools and McpxServlets, respectively.

To make tools available to Gemini, we will need to expose the schema of each tool as a FunctionDeclaration. The following FunctionRepository will provide a way to lookup the right declaration and invoke the right implementation by name.

package com.dylibso.mcpx4j.examples.gemini

import android.util.Log
import com.dylibso.mcpx4j.core.McpxTool
import com.google.ai.client.generativeai.type.FunctionDeclaration
import org.json.JSONObject

data class FunctionRepository(
    val functionDeclarations: Map<String, FunctionDeclaration>,
    val mcpxTools: Map<String, McpxTool>) {

    fun call(name: String, args: Map<String, String?>): JSONObject {
        val tool = mcpxTools[name]
            ?: return JSONObject(
                mapOf("result" to "$name is not a valid function"))

        val jargs = JSONObject(args)
        val jsargs = JSONObject(mapOf(
            "method" to "tools/call",
            "params" to mapOf(
                "name" to name,
                "arguments" to jargs
            )))

        Log.i("mcpx4j", "invoking $name with args = $jargs")
        // Invoke the mcp.run tool with the given arguments
        val res = tool.call(jsargs.toString())
        Log.i("mcpx4j", "$name returned: $res")

        // Ensure we always return a map
        return JSONObject(mapOf("result" to res))
    }
}

Fetching the Servlets​

Now that we have set up the FunctionRepository we need to interface with mcp.run to fetch the implementations.

The following code configures an MCPX4J client and it builds a FunctionDefinition for each tool in each installed servlet.

package com.dylibso.mcpx4j.examples.gemini

import com.dylibso.mcpx4j.core.*
import com.google.ai.client.generativeai.type.*
import org.extism.sdk.chicory.*

object ToolFetcher {
    fun fetchFunctions(): FunctionRepository {
        val mcpx =
            // Configure the MCP.RUN Session
            Mcpx.forApiKey(BuildConfig.mcpRunKey)
                .withServletOptions(
                    McpxServletOptions.builder()
                        // Setup an HTTP client compatible with Android
                        // on the Chicory runtime
                        .withChicoryHttpConfig(
                            HttpConfig.builder()
                                .withJsonCodec(JacksonJsonCodec())
                                .withClientAdapter(
                                    HttpUrlConnectionClientAdapter())
                                .build())
                        // Configure an alternative, Android-specific logger
                        .withChicoryLogger(AndroidLogger("mcpx4j-runtime"))
                        .build())
                // Configure also the MCPX4J HTTP client to use
                // the Android-compatible implementation
                .withHttpClientAdapter(HttpUrlConnectionClientAdapter())
                .build()

        // Refresh once the list of installations.
        // This can be also scheduled for periodic refresh.
        mcpx.refreshInstallations("default")
        val servlets = mcpx.servlets()

        // Extract the metadata of each `McpxTool` into a `FunctionDeclaration`
        val functionDeclarations =
            servlets.flatMap {
                it.tools().map { 
                    it.value.name() to toFunctionDeclaration(it.value) } }
                    .toMap()
        // Create a map name -> McpxTool for quicker lookup
        val mcpxTools =
            servlets.flatMap {
                it.tools().map { 
                    it.value.name() to it.value } }.toMap()
        return FunctionRepository(functionDeclarations, mcpxTools)
    }

    private fun toFunctionDeclaration(tool: McpxTool): FunctionDeclaration {
        val parsedSchema = ParsedSchema.parse(tool.inputSchema())
        val f = defineFunction(
            name = tool.name(),
            description = tool.description(),
            parameters = parsedSchema.parameters,
            requiredParameters = parsedSchema.requiredParameters
        )
        return f
    }
}

You will also need to define the object AndroidLogger. Here is a simple implementation:

package com.dylibso.mcpx4j.examples.gemini

import android.util.Log
import com.dylibso.chicory.log.Logger

class AndroidLogger(val tag: String) : Logger {
    override fun log(
        level: Logger.Level?, msg: String?, throwable: Throwable?) {
        Log.i(tag, msg, throwable)
    }
    override fun isLoggable(level: Logger.Level): Boolean = true
}

You will also have noticed that we need to implement ParsedSchema.parse(tool.inputSchema()). For brevity we omit the implementation of this object, refer to the online repo for the detailed implementation.

Exposing the GenerativeModel​

This is where we configure the Gemini chat. We choose the model, we set the API key for Gemini, we plug our tools, and setup a nice system prompt.

package com.dylibso.mcpx4j.examples.gemini

import com.google.ai.client.generativeai.GenerativeModel
import com.google.ai.client.generativeai.type.FunctionResponsePart
import com.google.ai.client.generativeai.type.Tool
import com.google.ai.client.generativeai.type.content

class ChatSession(private val functionRepository: FunctionRepository) {
    private val generativeModel = GenerativeModel(
        // Use Gemini 2.0
        modelName = "gemini-2.0-flash-exp",
        // Use the API Key we configured in local.properties
        apiKey = BuildConfig.apiKey,
        // Plug our tools
        tools = listOf(Tool(
            functionRepository.functionDeclarations.values.toList())),
        // Configure a useful system prompt.
        systemInstruction = content {
            text(
                """
You are a helpful AI assistant with access to various external tools and APIs. Your goal is to complete tasks thoroughly and autonomously by making full use of these tools. Here are your core operating principles:

1.  **Take initiative** - Don't wait for user permission to use tools. If a tool would help complete the task, use it immediately.
2.  **Chain multiple tools together** - Many tasks require multiple tool calls in sequence. Plan out and execute the full chain of calls needed to achieve the goal.
3.  **Handle errors gracefully** - If a tool call fails, try alternative approaches or tools rather than asking the user what to do.
4.  **Make reasonable assumptions** - When tool calls require parameters, use your best judgment to provide appropriate values rather than asking the user.
5.  **Show your work** - After completing tool calls, explain what you did and show relevant results, but focus on the final outcome the user wanted.
6.  **Be thorough** - Use tools repeatedly as needed until you're confident you've fully completed the task. Don't stop at partial solutions.

Your responses should focus on results rather than asking questions. Only ask the user for clarification if the task itself is unclear or impossible with the tools available.
"""
            )
        }
    )

    // Initiate a chat session
    private val chat = generativeModel.startChat()

    // Notice this function does I/O, hence the `suspend`.
    suspend fun send(prompt: String): String? {
        // Send the first chat message and wait for the response
        var response = chat.sendMessage(prompt)

        // Continue processing until there are no other function calls
        while (response.functionCalls.isNotEmpty()) {
            // For each function call, produce a response
            val parts = response.functionCalls.map {
                // Lookup the function in the repository and invoke it
                val toolResult = functionRepository.call(it.name, it.args)
                // Return the result for that specific function
                FunctionResponsePart(it.name, toolResult)
            }
            // Send back a message with all the `FunctionResponsePart`s
            // re-assign to `response`, and continue the loop.
            response = chat.sendMessage(
                content(role = "function") {
                    parts.forEach { part(it) }
                }
            )
        }
        
        // No more functions left to process, return the text response.
        return response.text
    }
}

Wiring the ChatSession into the ViewModel​

Open the BakingViewModel and replace the following lines

    private val generativeModel = GenerativeModel(
        modelName = "gemini-1.5-pro",
        apiKey = BuildConfig.apiKey
    )

with these instead:

    private lateinit var generativeModel: ChatSession

    init {
        _uiState.value = UiState.Loading
        viewModelScope.launch {
            withContext(Dispatchers.IO) {
                generativeModel = ChatSession(ToolFetcher.fetchFunctions())
                _uiState.value = UiState.Initial
            }
        }
    }

Now you can already start your application!

Testing the Integration​

Try this example prompt to test the tool chaining capability:

I made a list of places I'd like to visit. Can you give directions between them and find a guide online? give me a summary of the directions and of the guide https://gist.github.com/evacchi/64dc59754417e1554398c84c29a8bd43

The assistant will automatically:

  1. Use fetch to download the list of places in Paris
  2. Lookup directions between these places using google-maps (the tool might be invoked several times)
  3. Lookup a travel guide using brave-search
  4. Return a summary of the directions and some travel information.

You should see output similar to this (you can "touch" and drag the text at the bottom to read more)

This demonstrates how the assistant can chain multiple tools together without explicit instructions, showcasing the power of the integration.

Final Touches​

The application still shows pictures of baking goods. So, it does not really fit the purpose of giving travel directions. For this demo, let us replace the pictures of these baking goods with pictures of attractions in Paris.

Open BakingScreen.kt and replace the first few lines:

val images = arrayOf(
    // Image from https://commons.wikimedia.org/wiki/File:Tour_Eiffel_Wikimedia_Commons_(cropped).jpg
    R.drawable.tour_eiffel,
    // Image from https://en.wikipedia.org/wiki/File:Mona_Lisa,_by_Leonardo_da_Vinci,_from_C2RMF_retouched.jpg
    R.drawable.mona_lisa,
    // Image from https://commons.wikimedia.org/wiki/File:Basilique_du_Sacr%C3%A9-C%C5%93ur_de_Montmartre,_Paris_18e_140223_2.jpg
    R.drawable.montmartre,
)

And make sure to create the files

Finally update the strings in /app/src/main/res/values/strings.xml:

<resources>
    <string name="app_name">Mcpx4jGemini</string>
    <string name="action_go">Go</string>
    <string name="baking_title">Mcpx4j with Gemini</string>
    <string name="label_prompt">Prompt</string>
    <string name="prompt_placeholder">
        I would like to visit the place in the first picture,
        and then see the painting in the second picture.
        Can you give directions between them and find a guide online?
        Start from the first, and then go to the second.
        Give me a summary of the directions and of the guide</string>
    <string name="image1_description">Tour Eiffel</string>
    <string name="image2_description">Mona Lisa</string>
    <string name="image3_description">Montmartre</string>
    <string name="results_placeholder">(Results will appear here)</string>
</resources>

We have now also updated the default prompt to:

I would like to visit the place in the first picture,
and then see the painting in the second picture.
Can you give directions between them and find a guide online?
Start from the first, and then go to the second.
Give me a summary of the directions and of the guide

Let's run our application for the last time. You can "touch" and drag the text controller at the bottom to read more.

Support​

If you get stuck and need some help, please reach out! Visit our support page to learn how best to get in touch.